SPF-Portrait：迈向基于语义无污染微调的纯肖像定制

SPF-Portrait: Towards Pure Portrait Customization with Semantic Pollution-Free Fine-tuning

April 1, 2025

作者: Xiaole Xian, Zhichao Liao, Qingyu Li, Wenyu Qin, Pengfei Wan, Weicheng Xie, Long Zeng, Linlin Shen, Pingfa Feng

cs.AI

摘要

在定制肖像数据集上微调预训练的文本到图像（T2I）模型，是文本驱动肖像属性定制的主流方法。由于微调过程中的语义污染，现有方法在定制目标属性的同时，难以维持原始模型的行为并实现增量学习。为解决这一问题，我们提出了SPF-Portrait，这是一项开创性工作，旨在纯粹理解定制语义的同时，消除文本驱动肖像定制中的语义污染。在我们的SPF-Portrait中，我们提出了一种双路径管道，将原始模型作为传统微调路径的参考。通过对比学习，我们确保了对目标属性的适应，并有意将其他无关属性与原始肖像对齐。我们引入了一种新颖的语义感知精细控制图，它代表了目标语义的精确响应区域，以空间上指导对比路径之间的对齐过程。这一对齐过程不仅有效保留了原始模型的性能，还避免了过度对齐。此外，我们提出了一种新颖的响应增强机制，以强化目标属性的表现，同时缓解直接跨模态监督中固有的表示差异。大量实验证明，SPF-Portrait实现了最先进的性能。项目网页：https://spf-portrait.github.io/SPF-Portrait/

English

Fine-tuning a pre-trained Text-to-Image (T2I) model on a tailored portrait dataset is the mainstream method for text-driven customization of portrait attributes. Due to Semantic Pollution during fine-tuning, existing methods struggle to maintain the original model's behavior and achieve incremental learning while customizing target attributes. To address this issue, we propose SPF-Portrait, a pioneering work to purely understand customized semantics while eliminating semantic pollution in text-driven portrait customization. In our SPF-Portrait, we propose a dual-path pipeline that introduces the original model as a reference for the conventional fine-tuning path. Through contrastive learning, we ensure adaptation to target attributes and purposefully align other unrelated attributes with the original portrait. We introduce a novel Semantic-Aware Fine Control Map, which represents the precise response regions of the target semantics, to spatially guide the alignment process between the contrastive paths. This alignment process not only effectively preserves the performance of the original model but also avoids over-alignment. Furthermore, we propose a novel response enhancement mechanism to reinforce the performance of target attributes, while mitigating representation discrepancy inherent in direct cross-modal supervision. Extensive experiments demonstrate that SPF-Portrait achieves state-of-the-art performance. Project webpage: https://spf-portrait.github.io/SPF-Portrait/