StyleMaster：使用藝術生成和轉換為您的視頻添加風格

StyleMaster: Stylize Your Video with Artistic Generation and Translation

December 10, 2024

作者: Zixuan Ye, Huijuan Huang, Xintao Wang, Pengfei Wan, Di Zhang, Wenhan Luo

cs.AI

摘要

風格控制在影片生成模型中已廣受歡迎。現有方法通常生成與給定風格相去甚遠的影片，導致內容泄漏，並難以將一個影片轉換為所需風格。我們的第一個觀察是風格提取階段至關重要，而現有方法強調全局風格卻忽略了局部紋理。為了在保持風格的同時引入紋理特徵並防止內容泄漏，我們基於提示-補丁相似性過濾與內容相關的補丁，同時保留風格補丁；對於全局風格提取，我們通過模型幻覺生成成對風格數據集以促進對比學習，從而極大增強了絕對風格一致性。此外，為了彌補圖像到影片的差距，我們在靜態影片上訓練了一個輕量級運動適配器，隱式增強了風格化程度，使我們在圖像上訓練的模型能夠無縫應用於影片。得益於這些努力，我們的方法StyleMaster 不僅在風格相似度和時間一致性方面取得了顯著改善，還可以輕鬆推廣至具有灰瓦控制網絡的影片風格轉換。大量實驗和視覺化展示了StyleMaster明顯優於競爭對手，有效生成與文本內容一致並與參考圖像風格密切相符的高質量風格化影片。我們的項目頁面位於https://zixuan-ye.github.io/stylemaster。

English

Style control has been popular in video generation models. Existing methods often generate videos far from the given style, cause content leakage, and struggle to transfer one video to the desired style. Our first observation is that the style extraction stage matters, whereas existing methods emphasize global style but ignore local textures. In order to bring texture features while preventing content leakage, we filter content-related patches while retaining style ones based on prompt-patch similarity; for global style extraction, we generate a paired style dataset through model illusion to facilitate contrastive learning, which greatly enhances the absolute style consistency. Moreover, to fill in the image-to-video gap, we train a lightweight motion adapter on still videos, which implicitly enhances stylization extent, and enables our image-trained model to be seamlessly applied to videos. Benefited from these efforts, our approach, StyleMaster, not only achieves significant improvement in both style resemblance and temporal coherence, but also can easily generalize to video style transfer with a gray tile ControlNet. Extensive experiments and visualizations demonstrate that StyleMaster significantly outperforms competitors, effectively generating high-quality stylized videos that align with textual content and closely resemble the style of reference images. Our project page is at https://zixuan-ye.github.io/stylemaster