LVCD：基於參考的線條視頻著色與擴散模型

LVCD: Reference-based Lineart Video Colorization with Diffusion Models

September 19, 2024

作者: Zhitong Huang, Mohan Zhang, Jing Liao

cs.AI

摘要

我們提出了第一個基於參考線條著色的影片擴散框架。與先前僅依賴圖像生成模型逐幀著色線條的作品不同，我們的方法利用大規模預訓練的影片擴散模型來生成著色動畫影片。這種方法產生了更具時間一致性的結果，更能應對大範圍運動。首先，我們引入了Sketch-guided ControlNet，為影像到影片擴散模型提供額外控制，以進行可控影片合成，實現基於線條的動畫影片生成。然後，我們提出了Reference Attention，以促進從參考幀向包含快速和擴張運動的其他幀傳遞顏色。最後，我們提出了一種新的連續取樣方案，結合了Overlapped Blending Module和Prev-Reference Attention，以擴展影片擴散模型超越其原始固定長度限制，用於長影片著色。定性和定量結果均表明，我們的方法在幀和影片質量以及時間一致性方面顯著優於最先進的技術。此外，我們的方法能夠生成具有大範圍運動的高質量、長時間一致的動畫影片，這在先前的作品中是無法實現的。我們的代碼和模型可在https://luckyhzt.github.io/lvcd找到。

English

We propose the first video diffusion framework for reference-based lineart video colorization. Unlike previous works that rely solely on image generative models to colorize lineart frame by frame, our approach leverages a large-scale pretrained video diffusion model to generate colorized animation videos. This approach leads to more temporally consistent results and is better equipped to handle large motions. Firstly, we introduce Sketch-guided ControlNet which provides additional control to finetune an image-to-video diffusion model for controllable video synthesis, enabling the generation of animation videos conditioned on lineart. We then propose Reference Attention to facilitate the transfer of colors from the reference frame to other frames containing fast and expansive motions. Finally, we present a novel scheme for sequential sampling, incorporating the Overlapped Blending Module and Prev-Reference Attention, to extend the video diffusion model beyond its original fixed-length limitation for long video colorization. Both qualitative and quantitative results demonstrate that our method significantly outperforms state-of-the-art techniques in terms of frame and video quality, as well as temporal consistency. Moreover, our method is capable of generating high-quality, long temporal-consistent animation videos with large motions, which is not achievable in previous works. Our code and model are available at https://luckyhzt.github.io/lvcd.