ColorFlow：檢索增強影像序列著色

ColorFlow: Retrieval-Augmented Image Sequence Colorization

December 16, 2024

作者: Junhao Zhuang, Xuan Ju, Zhaoyang Zhang, Yong Liu, Shiyi Zhang, Chun Yuan, Ying Shan

cs.AI

摘要

在保留角色和物件身份(ID)的前提下自動對黑白影像序列進行著色是一項複雜的任務，市場對此有著顯著需求，例如在卡通或漫畫系列的著色中。儘管使用大規模生成模型（如擴散模型）在視覺著色方面取得了進展，但在可控性和身份一致性方面仍存在挑戰，使得目前的解決方案不適用於工業應用。為了應對這一問題，我們提出了ColorFlow，這是一個針對工業應用中影像序列著色的三階段擴散框架。與現有方法需要進行基於身份的微調或明確的身份嵌入提取不同，我們提出了一個新穎的堅固且通用的檢索增強著色流水線，用於對具有相關色彩參考的影像進行著色。我們的流水線還具有雙分支設計：一個分支用於色彩身份提取，另一個用於著色，充分利用擴散模型的優勢。我們利用擴散模型中的自我注意機制進行強大的上下文學習和色彩身份匹配。為了評估我們的模型，我們引入了ColorFlow-Bench，這是一個用於基於參考的著色的全面基準。結果顯示，ColorFlow在多個指標上優於現有模型，為連續影像著色設定了新的標準，並有可能造福藝術行業。我們在我們的項目頁面上發布了我們的代碼和模型：https://zhuang2002.github.io/ColorFlow/。

English

Automatic black-and-white image sequence colorization while preserving character and object identity (ID) is a complex task with significant market demand, such as in cartoon or comic series colorization. Despite advancements in visual colorization using large-scale generative models like diffusion models, challenges with controllability and identity consistency persist, making current solutions unsuitable for industrial application.To address this, we propose ColorFlow, a three-stage diffusion-based framework tailored for image sequence colorization in industrial applications. Unlike existing methods that require per-ID finetuning or explicit ID embedding extraction, we propose a novel robust and generalizable Retrieval Augmented Colorization pipeline for colorizing images with relevant color references. Our pipeline also features a dual-branch design: one branch for color identity extraction and the other for colorization, leveraging the strengths of diffusion models. We utilize the self-attention mechanism in diffusion models for strong in-context learning and color identity matching. To evaluate our model, we introduce ColorFlow-Bench, a comprehensive benchmark for reference-based colorization. Results show that ColorFlow outperforms existing models across multiple metrics, setting a new standard in sequential image colorization and potentially benefiting the art industry. We release our codes and models on our project page: https://zhuang2002.github.io/ColorFlow/.