LumiNet：潛在內在特性與擴散模型相遇，用於室內場景燈光重新照明

LumiNet: Latent Intrinsics Meets Diffusion Models for Indoor Scene Relighting

November 29, 2024

作者: Xiaoyan Xing, Konrad Groh, Sezer Karaoglu, Theo Gevers, Anand Bhattad

cs.AI

摘要

我們介紹了 LumiNet，一種新穎的架構，利用生成模型和潛在內在表示來進行有效的燈光轉移。給定一個源影像和一個目標照明影像，LumiNet 合成源場景的重新照明版本，捕捉目標照明。我們的方法做出了兩個關鍵貢獻：從基於 StyleGAN 的重新照明模型中提取數據的策略用於我們的訓練，以及一個修改的基於擴散的 ControlNet，處理來自源影像的潛在內在特性和來自目標影像的潛在外在特性。我們通過一個學習的適配器（MLP）進一步改進照明轉移，通過交叉注意力和微調注入目標的潛在外在特性。 與傳統的 ControlNet 不同，後者從單一場景生成帶有條件地圖的影像，LumiNet 從兩個不同影像處理潛在表示，保留源影像的幾何和反照率，同時從目標轉移照明特性。實驗表明，我們的方法成功地在具有不同空間佈局和材料的場景之間轉移複雜的照明現象，包括高光和間接照明，僅使用影像作為輸入，在具有挑戰性的室內場景上優於現有方法。

English

We introduce LumiNet, a novel architecture that leverages generative models and latent intrinsic representations for effective lighting transfer. Given a source image and a target lighting image, LumiNet synthesizes a relit version of the source scene that captures the target's lighting. Our approach makes two key contributions: a data curation strategy from the StyleGAN-based relighting model for our training, and a modified diffusion-based ControlNet that processes both latent intrinsic properties from the source image and latent extrinsic properties from the target image. We further improve lighting transfer through a learned adaptor (MLP) that injects the target's latent extrinsic properties via cross-attention and fine-tuning. Unlike traditional ControlNet, which generates images with conditional maps from a single scene, LumiNet processes latent representations from two different images - preserving geometry and albedo from the source while transferring lighting characteristics from the target. Experiments demonstrate that our method successfully transfers complex lighting phenomena including specular highlights and indirect illumination across scenes with varying spatial layouts and materials, outperforming existing approaches on challenging indoor scenes using only images as input.