在向量場中引導修正流模型以控制影像生成

摘要

擴散模型（DMs）在逼真度、圖像編輯和解決反問題方面表現出色，受益於無分類器指導和圖像反演技術。然而，矯正流模型（RFMs）對於這些任務仍未被充分探索。現有基於DM的方法通常需要額外訓練，缺乏對預訓練潛在模型的泛化能力，表現不佳，並且由於通過ODE求解器和反演過程的廣泛反向傳播而需要大量計算資源。在這項工作中，我們首先對RFMs的向量場動力學進行理論和實證研究，以有效引導去噪軌跡。我們的研究發現，我們可以以確定性和無梯度的方式導航向量場。利用這一特性，我們提出了FlowChef，利用向量場來引導去噪軌跡，用於受控圖像生成任務，並通過跳過梯度來實現。FlowChef是一個統一的框架，可同時解決分類器指導、線性反問題和圖像編輯，無需額外訓練、反演或密集的反向傳播。最後，我們進行了廣泛的評估，並展示FlowChef在性能、內存和時間需求方面顯著優於基準，實現了新的最先進結果。項目頁面：https://flowchef.github.io。

English

Diffusion models (DMs) excel in photorealism, image editing, and solving inverse problems, aided by classifier-free guidance and image inversion techniques. However, rectified flow models (RFMs) remain underexplored for these tasks. Existing DM-based methods often require additional training, lack generalization to pretrained latent models, underperform, and demand significant computational resources due to extensive backpropagation through ODE solvers and inversion processes. In this work, we first develop a theoretical and empirical understanding of the vector field dynamics of RFMs in efficiently guiding the denoising trajectory. Our findings reveal that we can navigate the vector field in a deterministic and gradient-free manner. Utilizing this property, we propose FlowChef, which leverages the vector field to steer the denoising trajectory for controlled image generation tasks, facilitated by gradient skipping. FlowChef is a unified framework for controlled image generation that, for the first time, simultaneously addresses classifier guidance, linear inverse problems, and image editing without the need for extra training, inversion, or intensive backpropagation. Finally, we perform extensive evaluations and show that FlowChef significantly outperforms baselines in terms of performance, memory, and time requirements, achieving new state-of-the-art results. Project Page: https://flowchef.github.io.

在向量場中引導修正流模型以控制影像生成

Steering Rectified Flow Models in the Vector Field for Controlled Image Generation

摘要

Summary

Support