IDArb：針對任意數量的輸入視圖和光線的內在分解。

IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminations

December 16, 2024

作者: Zhibing Li, Tong Wu, Jing Tan, Mengchen Zhang, Jiaqi Wang, Dahua Lin

cs.AI

摘要

從影像中捕捉幾何和材質信息仍然是計算機視覺和圖形學中的一個基本挑戰。傳統基於優化的方法通常需要數小時的計算時間，從密集的多視圖輸入中重建幾何、材質特性和環境照明，同時仍然面臨著照明和材料之間固有的模糊性。另一方面，基於學習的方法利用現有的3D對象數據集中豐富的材質先驗，但在保持多視圖一致性方面面臨挑戰。在本文中，我們介紹了IDArb，這是一個基於擴散的模型，旨在在不同照明條件下對任意數量的影像執行內在分解。我們的方法實現了對表面法線和材質特性的準確且多視圖一致的估計。這是通過一個新穎的跨視圖、跨域注意模塊和一種照明增強、視圖自適應的訓練策略實現的。此外，我們介紹了ARB-Objaverse，這是一個提供大規模多視圖內在數據和在不同照明條件下渲染的新數據集，支持強大的訓練。大量實驗表明，IDArb在質量和量化方面均優於最先進的方法。此外，我們的方法促進了一系列下游任務，包括單圖像燈光重繪、光度立體和3D重建，突顯了其在逼真的3D內容創作中的廣泛應用。

English

Capturing geometric and material information from images remains a fundamental challenge in computer vision and graphics. Traditional optimization-based methods often require hours of computational time to reconstruct geometry, material properties, and environmental lighting from dense multi-view inputs, while still struggling with inherent ambiguities between lighting and material. On the other hand, learning-based approaches leverage rich material priors from existing 3D object datasets but face challenges with maintaining multi-view consistency. In this paper, we introduce IDArb, a diffusion-based model designed to perform intrinsic decomposition on an arbitrary number of images under varying illuminations. Our method achieves accurate and multi-view consistent estimation on surface normals and material properties. This is made possible through a novel cross-view, cross-domain attention module and an illumination-augmented, view-adaptive training strategy. Additionally, we introduce ARB-Objaverse, a new dataset that provides large-scale multi-view intrinsic data and renderings under diverse lighting conditions, supporting robust training. Extensive experiments demonstrate that IDArb outperforms state-of-the-art methods both qualitatively and quantitatively. Moreover, our approach facilitates a range of downstream tasks, including single-image relighting, photometric stereo, and 3D reconstruction, highlighting its broad applications in realistic 3D content creation.