SmoothCache：一种用于扩散Transformer的通用推理加速技术

SmoothCache: A Universal Inference Acceleration Technique for Diffusion Transformers

November 15, 2024

作者: Joseph Liu, Joshua Geddes, Ziyu Guo, Haomiao Jiang, Mahesh Kumar Nandwana

cs.AI

摘要

扩散变压器（DiT）已成为强大的生成模型，可用于各种任务，包括图像、视频和语音合成。然而，由于需要重复评估资源密集型的注意力和前馈模块，其推理过程仍然计算昂贵。为解决这一问题，我们引入了SmoothCache，这是一种面向模型的推理加速技术，适用于DiT架构。SmoothCache利用观察到的在相邻扩散时间步之间的层输出之间的高相似性。通过分析来自小型校准集的逐层表示误差，SmoothCache在推理过程中自适应地缓存和重复使用关键特征。我们的实验证明，SmoothCache在保持甚至改善跨多种模态的生成质量的同时，实现了8%至71%的加速。我们展示了其在图像生成的DiT-XL、文本到视频的Open-Sora以及文本到音频的Stable Audio Open上的有效性，突显了其潜力，可以实现实时应用并扩大强大DiT模型的可访问性。

English

Diffusion Transformers (DiT) have emerged as powerful generative models for various tasks, including image, video, and speech synthesis. However, their inference process remains computationally expensive due to the repeated evaluation of resource-intensive attention and feed-forward modules. To address this, we introduce SmoothCache, a model-agnostic inference acceleration technique for DiT architectures. SmoothCache leverages the observed high similarity between layer outputs across adjacent diffusion timesteps. By analyzing layer-wise representation errors from a small calibration set, SmoothCache adaptively caches and reuses key features during inference. Our experiments demonstrate that SmoothCache achieves 8% to 71% speed up while maintaining or even improving generation quality across diverse modalities. We showcase its effectiveness on DiT-XL for image generation, Open-Sora for text-to-video, and Stable Audio Open for text-to-audio, highlighting its potential to enable real-time applications and broaden the accessibility of powerful DiT models.