神经光照: 利用多光扩散解锁准确的物体法线和材质估计

Neural LightRig: Unlocking Accurate Object Normal and Material Estimation with Multi-Light Diffusion

December 12, 2024

作者: Zexin He, Tengfei Wang, Xin Huang, Xingang Pan, Ziwei Liu

cs.AI

摘要

从单个图像中恢复对象的几何和材质是具有不完全约束性质的，因此具有挑战性。本文介绍了一种名为神经光照调控（Neural LightRig）的新颖框架，通过利用来自2D扩散先验的辅助多光照条件，提升内在估计能力。具体来说，1）我们首先利用大规模扩散模型中的光照先验，在具有专门设计的合成重照数据集上构建了我们的多光照扩散模型。该扩散模型生成多个一致的图像，每个图像由不同方向的点光源照明。2）通过利用这些多样的光照图像来减少估计不确定性，我们使用具有U-Net骨干的大型G-buffer模型进行训练，准确预测表面法线和材质。大量实验证实了我们的方法明显优于最先进的方法，实现了准确的表面法线和PBR材质估计，并具有生动的重照效果。代码和数据集可在我们的项目页面https://projects.zxhezexin.com/neural-lightrig 上获取。

English

Recovering the geometry and materials of objects from a single image is challenging due to its under-constrained nature. In this paper, we present Neural LightRig, a novel framework that boosts intrinsic estimation by leveraging auxiliary multi-lighting conditions from 2D diffusion priors. Specifically, 1) we first leverage illumination priors from large-scale diffusion models to build our multi-light diffusion model on a synthetic relighting dataset with dedicated designs. This diffusion model generates multiple consistent images, each illuminated by point light sources in different directions. 2) By using these varied lighting images to reduce estimation uncertainty, we train a large G-buffer model with a U-Net backbone to accurately predict surface normals and materials. Extensive experiments validate that our approach significantly outperforms state-of-the-art methods, enabling accurate surface normal and PBR material estimation with vivid relighting effects. Code and dataset are available on our project page at https://projects.zxhezexin.com/neural-lightrig.