TripoSG：使用大规模矫正流模型进行高保真度3D形状合成

TripoSG: High-Fidelity 3D Shape Synthesis using Large-Scale Rectified Flow Models

February 10, 2025

作者: Yangguang Li, Zi-Xin Zou, Zexiang Liu, Dehu Wang, Yuan Liang, Zhipeng Yu, Xingchao Liu, Yuan-Chen Guo, Ding Liang, Wanli Ouyang, Yan-Pei Cao

cs.AI

摘要

最近扩散技术的进步推动了图像和视频生成达到前所未有的质量水平，显著加速了生成式人工智能的部署和应用。然而，3D形状生成技术迄今仍然落后，受制于3D数据规模的限制、3D数据处理复杂性以及对3D领域先进技术的不足探索。当前的3D形状生成方法在输出质量、泛化能力和与输入条件的对齐方面面临重大挑战。我们提出了TripoSG，一种新的简化形状扩散范式，能够生成与输入图像精确对应的高保真度3D网格。具体来说，我们提出：1）一个用于3D形状生成的大规模矫正流变换器，通过在大量高质量数据上训练实现了最先进的保真度。2）一种混合监督训练策略，结合SDF、法线和埃克纳尔损失用于3D VAE，实现高质量的3D重建性能。3）一个数据处理流水线，生成200万个高质量3D样本，突显了在训练3D生成模型中数据质量和数量的关键规则。通过全面实验，我们验证了新框架中每个组件的有效性。这些部分的无缝集成使TripoSG在3D形状生成方面实现了最先进的性能。由于具有高分辨率能力，生成的3D形状展现出增强的细节，并且对输入图像表现出卓越的保真度。此外，TripoSG展示了在从不同图像风格和内容生成3D模型方面的改进多样性，展示了强大的泛化能力。为促进3D生成领域的进步和创新，我们将公开提供我们的模型。

English

Recent advancements in diffusion techniques have propelled image and video generation to unprece- dented levels of quality, significantly accelerating the deployment and application of generative AI. However, 3D shape generation technology has so far lagged behind, constrained by limitations in 3D data scale, complexity of 3D data process- ing, and insufficient exploration of advanced tech- niques in the 3D domain. Current approaches to 3D shape generation face substantial challenges in terms of output quality, generalization capa- bility, and alignment with input conditions. We present TripoSG, a new streamlined shape diffu- sion paradigm capable of generating high-fidelity 3D meshes with precise correspondence to input images. Specifically, we propose: 1) A large-scale rectified flow transformer for 3D shape generation, achieving state-of-the-art fidelity through training on extensive, high-quality data. 2) A hybrid supervised training strategy combining SDF, normal, and eikonal losses for 3D VAE, achieving high- quality 3D reconstruction performance. 3) A data processing pipeline to generate 2 million high- quality 3D samples, highlighting the crucial rules for data quality and quantity in training 3D gen- erative models. Through comprehensive experi- ments, we have validated the effectiveness of each component in our new framework. The seamless integration of these parts has enabled TripoSG to achieve state-of-the-art performance in 3D shape generation. The resulting 3D shapes exhibit en- hanced detail due to high-resolution capabilities and demonstrate exceptional fidelity to input im- ages. Moreover, TripoSG demonstrates improved versatility in generating 3D models from diverse image styles and contents, showcasing strong gen- eralization capabilities. To foster progress and innovation in the field of 3D generation, we will make our model publicly available.