神經光線燈: 透過多光擴散解鎖準確的物體法線和材質估計

Neural LightRig: Unlocking Accurate Object Normal and Material Estimation with Multi-Light Diffusion

December 12, 2024

作者: Zexin He, Tengfei Wang, Xin Huang, Xingang Pan, Ziwei Liu

cs.AI

摘要

從單張圖像中恢復物體的幾何形狀和材質是具有不完全約束性質的，因此具有挑戰性。本文提出了一個新穎的框架 Neural LightRig，通過利用來自 2D 擴散先驗的輔助多光照條件，提升內在估計能力。具體來說，1) 我們首先利用大規模擴散模型中的照明先驗，在具有專用設計的合成照明數據集上構建我們的多光照擴散模型。該擴散模型生成多個一致的圖像，每個圖像由不同方向的點光源照明。2) 通過使用這些不同照明的圖像來降低估計不確定性，我們訓練了一個帶有 U-Net 骨幹的大型 G-buffer 模型，以準確預測表面法線和材質。大量實驗驗證了我們的方法明顯優於最先進的方法，實現了準確的表面法線和 PBR 材質估計，並具有生動的照明效果。代碼和數據集可在我們的項目頁面上找到：https://projects.zxhezexin.com/neural-lightrig。

English

Recovering the geometry and materials of objects from a single image is challenging due to its under-constrained nature. In this paper, we present Neural LightRig, a novel framework that boosts intrinsic estimation by leveraging auxiliary multi-lighting conditions from 2D diffusion priors. Specifically, 1) we first leverage illumination priors from large-scale diffusion models to build our multi-light diffusion model on a synthetic relighting dataset with dedicated designs. This diffusion model generates multiple consistent images, each illuminated by point light sources in different directions. 2) By using these varied lighting images to reduce estimation uncertainty, we train a large G-buffer model with a U-Net backbone to accurately predict surface normals and materials. Extensive experiments validate that our approach significantly outperforms state-of-the-art methods, enabling accurate surface normal and PBR material estimation with vivid relighting effects. Code and dataset are available on our project page at https://projects.zxhezexin.com/neural-lightrig.