通過文本到圖像 RGBA 實例生成生成組合場景

Generating Compositional Scenes via Text-to-image RGBA Instance Generation

November 16, 2024

作者: Alessandro Fontanella, Petru-Daniel Tudosiu, Yongxin Yang, Shifeng Zhang, Sarah Parisot

cs.AI

摘要

文字到圖像擴散生成模型能夠生成高質量圖像，但需要耗費大量時間進行提示工程。通過引入佈局條件，可以提高可控性，然而現有方法缺乏佈局編輯能力和對象屬性的精細控制。多層生成的概念具有潛力來解決這些限制，然而同時生成圖像實例和場景構成會限制對對象屬性的精細控制、在3D空間中的相對位置和場景操作能力。在這項工作中，我們提出了一種新穎的多階段生成範式，旨在實現對對象屬性、靈活性和互動性的精細控制。為確保對實例屬性的控制，我們設計了一種新穎的訓練範式，以適應一個擴散模型，生成帶有透明信息的獨立場景組件作為RGBA圖像。為了構建復雜圖像，我們利用這些預生成的實例，並引入一個多層合成生成過程，平滑地將組件組裝在逼真的場景中。我們的實驗表明，我們的RGBA擴散模型能夠生成多樣且高質量的實例，並能精確控制對象屬性。通過多層合成，我們展示了我們的方法允許從高度復雜的提示中構建和操作圖像，對對象外觀和位置具有精細控制，比競爭方法具有更高程度的控制。

English

Text-to-image diffusion generative models can generate high quality images at the cost of tedious prompt engineering. Controllability can be improved by introducing layout conditioning, however existing methods lack layout editing ability and fine-grained control over object attributes. The concept of multi-layer generation holds great potential to address these limitations, however generating image instances concurrently to scene composition limits control over fine-grained object attributes, relative positioning in 3D space and scene manipulation abilities. In this work, we propose a novel multi-stage generation paradigm that is designed for fine-grained control, flexibility and interactivity. To ensure control over instance attributes, we devise a novel training paradigm to adapt a diffusion model to generate isolated scene components as RGBA images with transparency information. To build complex images, we employ these pre-generated instances and introduce a multi-layer composite generation process that smoothly assembles components in realistic scenes. Our experiments show that our RGBA diffusion model is capable of generating diverse and high quality instances with precise control over object attributes. Through multi-layer composition, we demonstrate that our approach allows to build and manipulate images from highly complex prompts with fine-grained control over object appearance and location, granting a higher degree of control than competing methods.