通过文本到图像的RGBA实例生成生成组合场景

Generating Compositional Scenes via Text-to-image RGBA Instance Generation

November 16, 2024

作者: Alessandro Fontanella, Petru-Daniel Tudosiu, Yongxin Yang, Shifeng Zhang, Sarah Parisot

cs.AI

摘要

文本到图像扩散生成模型可以生成高质量图像，但需要繁琐的提示工程。通过引入布局条件，可以改善可控性，然而现有方法缺乏布局编辑能力和对物体属性的精细控制。多层生成的概念有望解决这些限制，然而同时生成图像实例和场景构成限制了对物体属性的精细控制、在三维空间中的相对定位和场景操作能力。在这项工作中，我们提出了一种新颖的多阶段生成范式，旨在实现精细控制、灵活性和互动性。为了确保对实例属性的控制，我们设计了一种新颖的训练范式，以调整扩散模型，生成带有透明信息的独立场景组件作为RGBA图像。为了构建复杂图像，我们利用这些预生成的实例，并引入多层复合生成过程，平滑地将组件组装在逼真的场景中。我们的实验表明，我们的RGBA扩散模型能够生成多样且高质量的实例，并精确控制物体属性。通过多层合成，我们展示了我们的方法允许从高度复杂的提示构建和操作图像，对物体外观和位置具有精细控制，比竞争方法具有更高程度的控制。

English

Text-to-image diffusion generative models can generate high quality images at the cost of tedious prompt engineering. Controllability can be improved by introducing layout conditioning, however existing methods lack layout editing ability and fine-grained control over object attributes. The concept of multi-layer generation holds great potential to address these limitations, however generating image instances concurrently to scene composition limits control over fine-grained object attributes, relative positioning in 3D space and scene manipulation abilities. In this work, we propose a novel multi-stage generation paradigm that is designed for fine-grained control, flexibility and interactivity. To ensure control over instance attributes, we devise a novel training paradigm to adapt a diffusion model to generate isolated scene components as RGBA images with transparency information. To build complex images, we employ these pre-generated instances and introduce a multi-layer composite generation process that smoothly assembles components in realistic scenes. Our experiments show that our RGBA diffusion model is capable of generating diverse and high quality instances with precise control over object attributes. Through multi-layer composition, we demonstrate that our approach allows to build and manipulate images from highly complex prompts with fine-grained control over object appearance and location, granting a higher degree of control than competing methods.