TEXGen：一种用于网格纹理的生成扩散模型

TEXGen: a Generative Diffusion Model for Mesh Textures

November 22, 2024

作者: Xin Yu, Ze Yuan, Yuan-Chen Guo, Ying-Tian Liu, JianHui Liu, Yangguang Li, Yan-Pei Cao, Ding Liang, Xiaojuan Qi

cs.AI

摘要

高质量的纹理贴图对于逼真的3D资产渲染至关重要，然而鲜有研究直接探索在纹理空间中学习，尤其是在大规模数据集上。在这项工作中，我们摆脱了依赖预训练的2D扩散模型在测试时优化3D纹理的传统方法。相反，我们专注于在UV纹理空间本身学习的基本问题。我们首次训练了一个大型扩散模型，能够以前馈方式直接生成高分辨率的纹理贴图。为了促进在高分辨率UV空间中的高效学习，我们提出了一种可扩展的网络架构，交替在UV贴图上进行卷积，并在点云上使用注意力层。利用这种架构设计，我们训练了一个拥有7亿参数的扩散模型，可以生成由文本提示和单视图图像指导的UV纹理贴图。一经训练，我们的模型自然支持各种扩展应用，包括文本引导的纹理修复、稀疏视图纹理完成以及文本驱动的纹理合成。项目页面位于http://cvmi-lab.github.io/TEXGen/。

English

While high-quality texture maps are essential for realistic 3D asset rendering, few studies have explored learning directly in the texture space, especially on large-scale datasets. In this work, we depart from the conventional approach of relying on pre-trained 2D diffusion models for test-time optimization of 3D textures. Instead, we focus on the fundamental problem of learning in the UV texture space itself. For the first time, we train a large diffusion model capable of directly generating high-resolution texture maps in a feed-forward manner. To facilitate efficient learning in high-resolution UV spaces, we propose a scalable network architecture that interleaves convolutions on UV maps with attention layers on point clouds. Leveraging this architectural design, we train a 700 million parameter diffusion model that can generate UV texture maps guided by text prompts and single-view images. Once trained, our model naturally supports various extended applications, including text-guided texture inpainting, sparse-view texture completion, and text-driven texture synthesis. Project page is at http://cvmi-lab.github.io/TEXGen/.