ColorFlow: 检索增强型图像序列着色

ColorFlow: Retrieval-Augmented Image Sequence Colorization

December 16, 2024

作者: Junhao Zhuang, Xuan Ju, Zhaoyang Zhang, Yong Liu, Shiyi Zhang, Chun Yuan, Ying Shan

cs.AI

摘要

在保留角色和物体身份的前提下自动对黑白图像序列进行着色是一项具有重要市场需求的复杂任务，例如在卡通或漫画系列的着色中。尽管使用大规模生成模型（如扩散模型）在视觉着色方面取得了进展，但在可控性和身份一致性方面仍存在挑战，使当前解决方案不适用于工业应用。为了解决这一问题，我们提出了ColorFlow，这是一个专为工业应用中的图像序列着色而定制的三阶段扩散框架。与现有方法需要进行基于身份的微调或明确的身份嵌入提取不同，我们提出了一种新颖的强大且通用的检索增强着色管道，用于使用相关颜色参考对图像进行着色。我们的管道还采用了双分支设计：一个分支用于颜色身份提取，另一个用于着色，充分利用了扩散模型的优势。我们利用扩散模型中的自注意机制进行强大的上下文学习和颜色身份匹配。为了评估我们的模型，我们引入了ColorFlow-Bench，这是一个用于基于参考的着色的全面基准。结果显示，ColorFlow在多个指标上优于现有模型，为顺序图像着色设定了新标准，并有望使艺术行业受益。我们在项目页面上发布了我们的代码和模型：https://zhuang2002.github.io/ColorFlow/。

English

Automatic black-and-white image sequence colorization while preserving character and object identity (ID) is a complex task with significant market demand, such as in cartoon or comic series colorization. Despite advancements in visual colorization using large-scale generative models like diffusion models, challenges with controllability and identity consistency persist, making current solutions unsuitable for industrial application.To address this, we propose ColorFlow, a three-stage diffusion-based framework tailored for image sequence colorization in industrial applications. Unlike existing methods that require per-ID finetuning or explicit ID embedding extraction, we propose a novel robust and generalizable Retrieval Augmented Colorization pipeline for colorizing images with relevant color references. Our pipeline also features a dual-branch design: one branch for color identity extraction and the other for colorization, leveraging the strengths of diffusion models. We utilize the self-attention mechanism in diffusion models for strong in-context learning and color identity matching. To evaluate our model, we introduce ColorFlow-Bench, a comprehensive benchmark for reference-based colorization. Results show that ColorFlow outperforms existing models across multiple metrics, setting a new standard in sequential image colorization and potentially benefiting the art industry. We release our codes and models on our project page: https://zhuang2002.github.io/ColorFlow/.