SynthLight：透過學習重新渲染合成臉部的擴散模型來進行肖像燈光調整。

SynthLight: Portrait Relighting with Diffusion Model by Learning to Re-render Synthetic Faces

January 16, 2025

作者: Sumit Chaturvedi, Mengwei Ren, Yannick Hold-Geoffroy, Jingyuan Liu, Julie Dorsey, Zhixin Shu

cs.AI

摘要

我們介紹了 SynthLight，一種用於人像燈光重繪的擴散模型。我們的方法將圖像燈光重繪定義為一個重新渲染問題，其中像素會根據環境燈光條件的變化而進行轉換。利用基於物理的渲染引擎，我們合成了一個數據集，以模擬在不同照明下對3D頭部資產進行這種照明條件轉換。我們提出了兩種訓練和推斷策略，以彌合合成和真實圖像領域之間的差距：(1) 多任務訓練，利用沒有照明標籤的真實人像；(2) 基於無分類器指導的推斷時間擴散採樣程序，利用輸入人像以更好地保留細節。我們的方法推廣到多樣的真實照片，產生逼真的照明效果，包括鏡面高光和投影陰影，同時保留主題的身份。我們在 Light Stage 數據上的定量實驗表明，我們的結果與最先進的燈光重繪方法相當。我們對野外圖像的定性結果展示了豐富且前所未有的照明效果。項目頁面：https://vrroom.github.io/synthlight/

English

We introduce SynthLight, a diffusion model for portrait relighting. Our approach frames image relighting as a re-rendering problem, where pixels are transformed in response to changes in environmental lighting conditions. Using a physically-based rendering engine, we synthesize a dataset to simulate this lighting-conditioned transformation with 3D head assets under varying lighting. We propose two training and inference strategies to bridge the gap between the synthetic and real image domains: (1) multi-task training that takes advantage of real human portraits without lighting labels; (2) an inference time diffusion sampling procedure based on classifier-free guidance that leverages the input portrait to better preserve details. Our method generalizes to diverse real photographs and produces realistic illumination effects, including specular highlights and cast shadows, while preserving the subject's identity. Our quantitative experiments on Light Stage data demonstrate results comparable to state-of-the-art relighting methods. Our qualitative results on in-the-wild images showcase rich and unprecedented illumination effects. Project Page: https://vrroom.github.io/synthlight/