MaterialFusion：利用材料擴散增強反渲染先驗

MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors

September 23, 2024

作者: Yehonathan Litman, Or Patashnik, Kangle Deng, Aviral Agrawal, Rushikesh Zawar, Fernando De la Torre, Shubham Tulsiani

cs.AI

摘要

最近在反渲染方面的研究表明，使用物体的多视图图像恢复形状、反照率和材质具有潜力。然而，由于从输入图像中分离反照率和材质属性的固有挑战，恢复的组件通常无法在新的光照条件下准确渲染。为了解决这一挑战，我們引入了MaterialFusion，这是一种增强的传统3D反渲染流程，结合了对纹理和材质属性的2D先验。我们提出了StableMaterial，这是一个2D扩散模型先验，可以优化多光照数据，从给定的输入外观中估计最可能的反照率和材质。该模型是在一个由约12K个艺术家设计的合成Blender对象组成的策划数据集BlenderVault上训练的，其中包括反照率、材质和重新照明图像数据。我们将这种扩散先验与反渲染框架相结合，使用得分蒸馏采样（SDS）来引导反照率和材质的优化，提高了与先前工作相比的重新照明性能。我们验证了MaterialFusion在4个合成和真实对象数据集上在不同照明条件下的重新照明性能，展示了我们的扩散辅助方法显著改善了在新的光照条件下重建对象的外观。我们打算公开发布我们的BlenderVault数据集，以支持这一领域的进一步研究。

English

Recent works in inverse rendering have shown promise in using multi-view images of an object to recover shape, albedo, and materials. However, the recovered components often fail to render accurately under new lighting conditions due to the intrinsic challenge of disentangling albedo and material properties from input images. To address this challenge, we introduce MaterialFusion, an enhanced conventional 3D inverse rendering pipeline that incorporates a 2D prior on texture and material properties. We present StableMaterial, a 2D diffusion model prior that refines multi-lit data to estimate the most likely albedo and material from given input appearances. This model is trained on albedo, material, and relit image data derived from a curated dataset of approximately ~12K artist-designed synthetic Blender objects called BlenderVault. we incorporate this diffusion prior with an inverse rendering framework where we use score distillation sampling (SDS) to guide the optimization of the albedo and materials, improving relighting performance in comparison with previous work. We validate MaterialFusion's relighting performance on 4 datasets of synthetic and real objects under diverse illumination conditions, showing our diffusion-aided approach significantly improves the appearance of reconstructed objects under novel lighting conditions. We intend to publicly release our BlenderVault dataset to support further research in this field.