使用多光照合成的擴散方法進行輻射場照明調整

A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesis

September 13, 2024

作者: Yohan Poirier-Ginter, Alban Gauthier, Julien Phillip, Jean-Francois Lalonde, George Drettakis

cs.AI

摘要

對於多視角資料，重新照明輻射場存在嚴重的不完全約束，因為大多數情況下是在單一照明條件下捕獲的；尤其對於包含多個物體的完整場景更加困難。我們提出了一種方法，通過利用從2D圖像擴散模型中提取的先驗信息，使用這種單一照明資料創建可重新照明的輻射場。我們首先在一個以光線方向為條件的多照明資料集上對2D擴散模型進行微調，這使我們能夠將單一照明捕獲轉換為一個逼真但可能不一致的多照明資料集，其中光線方向是直接定義的。我們使用這些擴增資料來創建由3D高斯斑點表示的可重新照明的輻射場。為了實現對低頻照明的直接控制，我們使用一個以光線方向為參數的多層感知器來表示外觀。為了強制實現多視角一致性並克服不準確性，我們優化了每個圖像的輔助特徵向量。我們展示了在單一照明下的合成和真實多視角資料上的結果，證明了我們的方法成功地利用2D擴散模型的先驗信息，實現了對完整場景進行逼真的3D重新照明。項目網站：https://repo-sam.inria.fr/fungraph/generative-radiance-field-relighting/

English

Relighting radiance fields is severely underconstrained for multi-view data, which is most often captured under a single illumination condition; It is especially hard for full scenes containing multiple objects. We introduce a method to create relightable radiance fields using such single-illumination data by exploiting priors extracted from 2D image diffusion models. We first fine-tune a 2D diffusion model on a multi-illumination dataset conditioned by light direction, allowing us to augment a single-illumination capture into a realistic -- but possibly inconsistent -- multi-illumination dataset from directly defined light directions. We use this augmented data to create a relightable radiance field represented by 3D Gaussian splats. To allow direct control of light direction for low-frequency lighting, we represent appearance with a multi-layer perceptron parameterized on light direction. To enforce multi-view consistency and overcome inaccuracies we optimize a per-image auxiliary feature vector. We show results on synthetic and real multi-view data under single illumination, demonstrating that our method successfully exploits 2D diffusion model priors to allow realistic 3D relighting for complete scenes. Project site https://repo-sam.inria.fr/fungraph/generative-radiance-field-relighting/