在矢量场中引导修正流模型以控制图像生成

摘要

扩散模型（DMs）在逼真度、图像编辑和解决反问题方面表现出色，得益于无分类器指导和图像反演技术。然而，矫正流模型（RFMs）在这些任务中仍未得到充分探索。现有基于DM的方法通常需要额外训练，缺乏对预训练潜在模型的泛化能力，性能不佳，并且由于通过ODE求解器和反演过程的广泛反向传播而需要大量计算资源。在这项工作中，我们首先对RFMs的矢量场动力学进行理论和实证研究，以有效引导去噪轨迹。我们的研究结果显示，我们可以以确定性和无梯度的方式导航矢量场。利用这一特性，我们提出了FlowChef，利用矢量场引导去噪轨迹进行受控图像生成任务，通过跳过梯度实现，FlowChef是一个统一的框架，首次同时解决分类器指导、线性反问题和图像编辑，无需额外训练、反演或密集反向传播。最后，我们进行了广泛评估，并展示FlowChef在性能、内存和时间需求方面明显优于基线方法，取得了新的最先进结果。项目页面：https://flowchef.github.io。

English

Diffusion models (DMs) excel in photorealism, image editing, and solving inverse problems, aided by classifier-free guidance and image inversion techniques. However, rectified flow models (RFMs) remain underexplored for these tasks. Existing DM-based methods often require additional training, lack generalization to pretrained latent models, underperform, and demand significant computational resources due to extensive backpropagation through ODE solvers and inversion processes. In this work, we first develop a theoretical and empirical understanding of the vector field dynamics of RFMs in efficiently guiding the denoising trajectory. Our findings reveal that we can navigate the vector field in a deterministic and gradient-free manner. Utilizing this property, we propose FlowChef, which leverages the vector field to steer the denoising trajectory for controlled image generation tasks, facilitated by gradient skipping. FlowChef is a unified framework for controlled image generation that, for the first time, simultaneously addresses classifier guidance, linear inverse problems, and image editing without the need for extra training, inversion, or intensive backpropagation. Finally, we perform extensive evaluations and show that FlowChef significantly outperforms baselines in terms of performance, memory, and time requirements, achieving new state-of-the-art results. Project Page: https://flowchef.github.io.

在矢量场中引导修正流模型以控制图像生成

Steering Rectified Flow Models in the Vector Field for Controlled Image Generation

摘要

Summary

Support

Support