StyleMaster：使用艺术生成和转换为您的视频添加风格

StyleMaster: Stylize Your Video with Artistic Generation and Translation

December 10, 2024

作者: Zixuan Ye, Huijuan Huang, Xintao Wang, Pengfei Wan, Di Zhang, Wenhan Luo

cs.AI

摘要

风格控制在视频生成模型中已经很受欢迎。现有方法通常生成与给定风格相去甚远的视频，导致内容泄漏，并且难以将一个视频转换为所需风格。我们的第一个观察是风格提取阶段很重要，而现有方法强调全局风格却忽略了局部纹理。为了在保持风格的同时引入纹理特征并防止内容泄漏，我们基于提示-补丁相似性过滤与内容相关的补丁，同时保留风格补丁；对于全局风格提取，我们通过模型幻觉生成一对风格数据集，以促进对比学习，从而极大地增强了绝对风格一致性。此外，为了弥补图像到视频的差距，我们在静态视频上训练了一个轻量级运动适配器，隐式增强了风格化程度，并使我们在图像上训练的模型能够无缝地应用于视频。得益于这些努力，我们的方法StyleMaster 不仅在风格相似度和时间上的连贯性方面取得了显著改进，而且可以轻松推广到视频风格转移，使用了灰瓦控制网络。大量实验和可视化展示表明，StyleMaster 明显优于竞争对手，有效生成与文本内容一致且与参考图像风格紧密匹配的高质量风格化视频。我们的项目页面位于 https://zixuan-ye.github.io/stylemaster。

English

Style control has been popular in video generation models. Existing methods often generate videos far from the given style, cause content leakage, and struggle to transfer one video to the desired style. Our first observation is that the style extraction stage matters, whereas existing methods emphasize global style but ignore local textures. In order to bring texture features while preventing content leakage, we filter content-related patches while retaining style ones based on prompt-patch similarity; for global style extraction, we generate a paired style dataset through model illusion to facilitate contrastive learning, which greatly enhances the absolute style consistency. Moreover, to fill in the image-to-video gap, we train a lightweight motion adapter on still videos, which implicitly enhances stylization extent, and enables our image-trained model to be seamlessly applied to videos. Benefited from these efforts, our approach, StyleMaster, not only achieves significant improvement in both style resemblance and temporal coherence, but also can easily generalize to video style transfer with a gray tile ControlNet. Extensive experiments and visualizations demonstrate that StyleMaster significantly outperforms competitors, effectively generating high-quality stylized videos that align with textual content and closely resemble the style of reference images. Our project page is at https://zixuan-ye.github.io/stylemaster