去噪重複使用：利用幀間運動一致性進行高效視頻潛在生成

Denoising Reuse: Exploiting Inter-frame Motion Consistency for Efficient Video Latent Generation

September 19, 2024

作者: Chenyu Wang, Shuo Yan, Yixuan Chen, Yujiang Wang, Mingzhi Dong, Xiaochen Yang, Dongsheng Li, Robert P. Dick, Qin Lv, Fan Yang, Tun Lu, Ning Gu, Li Shang

cs.AI

摘要

基於擴散的模型進行視頻生成受到高計算成本的限制，這是由於逐幀迭代擴散過程所致。本研究提出了一個名為Diffusion Reuse MOtion（Dr. Mo）網絡，用於加速潛在的視頻生成。我們的關鍵發現是，在早期去噪步驟中的粗粒度噪聲展現出在連續視頻幀中高運動一致性。根據這一觀察結果，Dr. Mo通過納入精心設計的輕量級幀間運動，將這些粗粒度噪聲傳播到下一幀，從而消除了逐幀擴散模型中的大量計算冗余。更敏感和細粒度的噪聲仍然通過後續的去噪步驟獲取，這對於保留視覺質量可能是至關重要的。因此，決定哪些中間步驟應該從基於運動的傳播轉換為去噪，可能是一個關鍵問題，也是效率和質量之間的關鍵折衷。Dr. Mo採用一個名為Denoising Step Selector（DSS）的元網絡，動態確定視頻幀中的理想中間步驟。對視頻生成和編輯任務的廣泛評估表明，Dr. Mo能夠顯著加速擴散模型在視頻任務中，同時提高視覺質量。

English

Video generation using diffusion-based models is constrained by high computational costs due to the frame-wise iterative diffusion process. This work presents a Diffusion Reuse MOtion (Dr. Mo) network to accelerate latent video generation. Our key discovery is that coarse-grained noises in earlier denoising steps have demonstrated high motion consistency across consecutive video frames. Following this observation, Dr. Mo propagates those coarse-grained noises onto the next frame by incorporating carefully designed, lightweight inter-frame motions, eliminating massive computational redundancy in frame-wise diffusion models. The more sensitive and fine-grained noises are still acquired via later denoising steps, which can be essential to retain visual qualities. As such, deciding which intermediate steps should switch from motion-based propagations to denoising can be a crucial problem and a key tradeoff between efficiency and quality. Dr. Mo employs a meta-network named Denoising Step Selector (DSS) to dynamically determine desirable intermediate steps across video frames. Extensive evaluations on video generation and editing tasks have shown that Dr. Mo can substantially accelerate diffusion models in video tasks with improved visual qualities.