IDArb：任意数量的输入视图和光照的内在分解

IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminations

December 16, 2024

作者: Zhibing Li, Tong Wu, Jing Tan, Mengchen Zhang, Jiaqi Wang, Dahua Lin

cs.AI

摘要

从图像中捕获几何和材质信息仍然是计算机视觉和图形学中的一个基本挑战。传统的基于优化的方法通常需要数小时的计算时间，从密集的多视角输入中重建几何、材质属性和环境光照，同时仍然在光照和材质之间的固有歧义方面存在困难。另一方面，基于学习的方法利用现有3D对象数据集中丰富的材质先验，但在保持多视角一致性方面面临挑战。在本文中，我们介绍了IDArb，这是一个基于扩散的模型，旨在对在不同照明条件下的任意数量图像执行内在分解。我们的方法实现了对表面法线和材质属性的准确和多视角一致的估计。这得益于一种新颖的跨视图、跨域注意力模块和一种照明增强、视角自适应的训练策略。此外，我们介绍了ARB-Objaverse，这是一个提供大规模多视角内在数据和在不同光照条件下渲染的新数据集，支持强大的训练。大量实验证明，IDArb在质量和数量上均优于现有方法。此外，我们的方法促进了一系列下游任务，包括单图像重照、光度立体和3D重建，突显了其在逼真3D内容创建中的广泛应用。

English

Capturing geometric and material information from images remains a fundamental challenge in computer vision and graphics. Traditional optimization-based methods often require hours of computational time to reconstruct geometry, material properties, and environmental lighting from dense multi-view inputs, while still struggling with inherent ambiguities between lighting and material. On the other hand, learning-based approaches leverage rich material priors from existing 3D object datasets but face challenges with maintaining multi-view consistency. In this paper, we introduce IDArb, a diffusion-based model designed to perform intrinsic decomposition on an arbitrary number of images under varying illuminations. Our method achieves accurate and multi-view consistent estimation on surface normals and material properties. This is made possible through a novel cross-view, cross-domain attention module and an illumination-augmented, view-adaptive training strategy. Additionally, we introduce ARB-Objaverse, a new dataset that provides large-scale multi-view intrinsic data and renderings under diverse lighting conditions, supporting robust training. Extensive experiments demonstrate that IDArb outperforms state-of-the-art methods both qualitatively and quantitatively. Moreover, our approach facilitates a range of downstream tasks, including single-image relighting, photometric stereo, and 3D reconstruction, highlighting its broad applications in realistic 3D content creation.