ConsisLoRA：提升基于LoRA风格迁移的内容与风格一致性

ConsisLoRA: Enhancing Content and Style Consistency for LoRA-based Style Transfer

March 13, 2025

作者: Bolin Chen, Baoquan Zhao, Haoran Xie, Yi Cai, Qing Li, Xudong Mao

cs.AI

摘要

风格迁移涉及将参考图像的风格转移到目标图像的内容上。基于LoRA（低秩适应）方法的最新进展在有效捕捉单幅图像风格方面展现出潜力。然而，这些方法仍面临内容不一致、风格错位和内容泄露等重大挑战。本文全面分析了在风格迁移背景下，标准扩散参数化（即学习预测噪声）的局限性。为解决这些问题，我们提出了ConsisLoRA，一种基于LoRA的方法，通过优化LoRA权重以预测原始图像而非噪声，从而增强内容和风格的一致性。我们还提出了一种两步训练策略，将内容与参考图像风格的学习解耦。为了有效捕捉内容图像的全局结构和局部细节，我们引入了逐步损失过渡策略。此外，我们提出了一种推理引导方法，可在推理过程中实现对内容和风格强度的连续控制。通过定性和定量评估，我们的方法在内容和风格一致性方面显示出显著提升，同时有效减少了内容泄露。

English

Style transfer involves transferring the style from a reference image to the content of a target image. Recent advancements in LoRA-based (Low-Rank Adaptation) methods have shown promise in effectively capturing the style of a single image. However, these approaches still face significant challenges such as content inconsistency, style misalignment, and content leakage. In this paper, we comprehensively analyze the limitations of the standard diffusion parameterization, which learns to predict noise, in the context of style transfer. To address these issues, we introduce ConsisLoRA, a LoRA-based method that enhances both content and style consistency by optimizing the LoRA weights to predict the original image rather than noise. We also propose a two-step training strategy that decouples the learning of content and style from the reference image. To effectively capture both the global structure and local details of the content image, we introduce a stepwise loss transition strategy. Additionally, we present an inference guidance method that enables continuous control over content and style strengths during inference. Through both qualitative and quantitative evaluations, our method demonstrates significant improvements in content and style consistency while effectively reducing content leakage.