扩散变压器的局部相关自适应正则化

In-Context LoRA for Diffusion Transformers

October 31, 2024

作者: Lianghua Huang, Wei Wang, Zhi-Fan Wu, Yupeng Shi, Huanzhang Dou, Chen Liang, Yutong Feng, Yu Liu, Jingren Zhou

cs.AI

摘要

最近的研究arXiv:2410.15027探讨了扩散变压器（DiTs）在任务不可知图像生成中的应用，方法是简单地通过在图像间连接注意力标记。然而，尽管使用了大量计算资源，生成图像的保真度仍然不理想。在本研究中，我们重新评估并简化了这一框架，假设文本到图像的DiTs固有地具有上下文生成能力，只需要进行最少的调整来激活它们。通过多样的任务实验，我们定性地证明了现有的文本到图像的DiTs可以有效地进行上下文生成而无需任何调整。基于这一观点，我们提出了一个非常简单的流程来利用DiTs的上下文能力：（1）连接图像而不是标记，（2）对多个图像进行联合字幕，（3）使用小数据集（例如20sim 100个样本）而不是使用大数据集进行全参数调整，应用任务特定的LoRA调整。我们将我们的模型命名为In-Context LoRA（IC-LoRA）。这种方法不需要对原始DiT模型进行任何修改，只需更改训练数据。显著地，我们的流程生成了更符合提示的高保真度图像集。虽然在调整数据方面是任务特定的，但我们的框架在体系结构和流程上仍然是任务不可知的，为社区提供了一个强大的工具，并为进一步研究产品级任务不可知生成系统提供了宝贵的见解。我们在https://github.com/ali-vilab/In-Context-LoRA发布了我们的代码、数据和模型。

English

Recent research arXiv:2410.15027 has explored the use of diffusion transformers (DiTs) for task-agnostic image generation by simply concatenating attention tokens across images. However, despite substantial computational resources, the fidelity of the generated images remains suboptimal. In this study, we reevaluate and streamline this framework by hypothesizing that text-to-image DiTs inherently possess in-context generation capabilities, requiring only minimal tuning to activate them. Through diverse task experiments, we qualitatively demonstrate that existing text-to-image DiTs can effectively perform in-context generation without any tuning. Building on this insight, we propose a remarkably simple pipeline to leverage the in-context abilities of DiTs: (1) concatenate images instead of tokens, (2) perform joint captioning of multiple images, and (3) apply task-specific LoRA tuning using small datasets (e.g., 20sim 100 samples) instead of full-parameter tuning with large datasets. We name our models In-Context LoRA (IC-LoRA). This approach requires no modifications to the original DiT models, only changes to the training data. Remarkably, our pipeline generates high-fidelity image sets that better adhere to prompts. While task-specific in terms of tuning data, our framework remains task-agnostic in architecture and pipeline, offering a powerful tool for the community and providing valuable insights for further research on product-level task-agnostic generation systems. We release our code, data, and models at https://github.com/ali-vilab/In-Context-LoRA