MARVEL-40M+: 多层次视觉阐释用于高保真度文本到3D内容创作

MARVEL-40M+: Multi-Level Visual Elaboration for High-Fidelity Text-to-3D Content Creation

November 26, 2024

作者: Sankalp Sinha, Mohammad Sadil Khan, Muhammad Usama, Shino Sam, Didier Stricker, Sk Aziz Ali, Muhammad Zeshan Afzal

cs.AI

摘要

从文本提示生成高保真度的3D内容仍然是计算机视觉中的一个重要挑战，原因在于现有数据集的规模、多样性和注释深度有限。为了解决这个问题，我们引入了MARVEL-40M+，这是一个包含4000万文本注释的庞大数据集，涵盖了来自七个主要3D数据集的超过890万个3D资产。我们的贡献在于引入了一种新颖的多阶段注释流程，该流程整合了开源预训练的多视图VLMs和LLMs，自动生成从详细（150-200个词）到简洁语义标签（10-20个词）的多层描述。这种结构既支持细粒度的3D重建，又支持快速原型设计。此外，我们将源数据集中的人类元数据纳入我们的注释流程中，以在注释中添加领域特定信息，并减少VLM的幻觉。此外，我们开发了MARVEL-FX3D，一个两阶段的文本到3D流程。我们使用我们的注释对Stable Diffusion进行微调，并使用预训练的图像到3D网络在15秒内生成3D纹理网格。广泛的评估表明，MARVEL-40M+在注释质量和语言多样性方面明显优于现有数据集，GPT-4的胜率为72.41%，人类评估者的胜率为73.40%。

English

Generating high-fidelity 3D content from text prompts remains a significant challenge in computer vision due to the limited size, diversity, and annotation depth of the existing datasets. To address this, we introduce MARVEL-40M+, an extensive dataset with 40 million text annotations for over 8.9 million 3D assets aggregated from seven major 3D datasets. Our contribution is a novel multi-stage annotation pipeline that integrates open-source pretrained multi-view VLMs and LLMs to automatically produce multi-level descriptions, ranging from detailed (150-200 words) to concise semantic tags (10-20 words). This structure supports both fine-grained 3D reconstruction and rapid prototyping. Furthermore, we incorporate human metadata from source datasets into our annotation pipeline to add domain-specific information in our annotation and reduce VLM hallucinations. Additionally, we develop MARVEL-FX3D, a two-stage text-to-3D pipeline. We fine-tune Stable Diffusion with our annotations and use a pretrained image-to-3D network to generate 3D textured meshes within 15s. Extensive evaluations show that MARVEL-40M+ significantly outperforms existing datasets in annotation quality and linguistic diversity, achieving win rates of 72.41% by GPT-4 and 73.40% by human evaluators.