HelloMeme：将空间编织关注集成到扩散模型中，嵌入高级和丰富保真度条件

HelloMeme: Integrating Spatial Knitting Attentions to Embed High-Level and Fidelity-Rich Conditions in Diffusion Models

October 30, 2024

作者: Shengkai Zhang, Nianhong Jiao, Tian Li, Chaojie Yang, Chenhui Xue, Boya Niu, Jun Gao

cs.AI

摘要

我们提出了一种有效的方法，用于将适配器插入文本到图像基础模型中，从而实现复杂下游任务的执行，同时保持基础模型的泛化能力。该方法的核心思想是优化与2D特征图相关的注意力机制，从而增强适配器的性能。这种方法在模因视频生成任务上得到了验证，并取得了显著的结果。我们希望这项工作能为大型文本到图像模型的后训练任务提供启示。此外，由于该方法展示出与SD1.5衍生模型良好的兼容性，对开源社区具有一定价值。因此，我们将发布相关代码（https://songkey.github.io/hellomeme）。

English

We propose an effective method for inserting adapters into text-to-image foundation models, which enables the execution of complex downstream tasks while preserving the generalization ability of the base model. The core idea of this method is to optimize the attention mechanism related to 2D feature maps, which enhances the performance of the adapter. This approach was validated on the task of meme video generation and achieved significant results. We hope this work can provide insights for post-training tasks of large text-to-image models. Additionally, as this method demonstrates good compatibility with SD1.5 derivative models, it holds certain value for the open-source community. Therefore, we will release the related code (https://songkey.github.io/hellomeme).