SpotLight：通过扩散实现阴影引导的物体重照。

SpotLight: Shadow-Guided Object Relighting via Diffusion

November 27, 2024

作者: Frédéric Fortier-Chouinard, Zitian Zhang, Louis-Etienne Messier, Mathieu Garon, Anand Bhattad, Jean-François Lalonde

cs.AI

摘要

最近的研究表明扩散模型可以作为强大的神经渲染引擎，可用于将虚拟对象插入图像中。然而，与典型基于物理的渲染器不同，神经渲染引擎受到对光照设置的手动控制能力的限制，而这通常对改善或个性化所需的图像结果至关重要。在本文中，我们展示了通过简单指定对象的期望阴影，可以实现对对象重新照明的精确控制。令人惊讶的是，我们表明仅将对象的阴影注入预先训练的基于扩散的神经渲染器，即可使其根据期望的光源位置准确着色对象，并在目标背景图像中正确协调对象（及其阴影）。我们的方法SpotLight 利用现有的神经渲染方法，实现了可控的重新照明结果，无需额外训练。具体来说，我们展示了它在最近文献中的两个神经渲染器上的应用。我们展示了SpotLight 在对象合成结果方面取得了优越的成果，无论是在数量上还是在感知上，都得到了用户研究的确认，胜过了专门设计用于重新照明的现有扩散模型。

English

Recent work has shown that diffusion models can be used as powerful neural rendering engines that can be leveraged for inserting virtual objects into images. Unlike typical physics-based renderers, however, neural rendering engines are limited by the lack of manual control over the lighting setup, which is often essential for improving or personalizing the desired image outcome. In this paper, we show that precise lighting control can be achieved for object relighting simply by specifying the desired shadows of the object. Rather surprisingly, we show that injecting only the shadow of the object into a pre-trained diffusion-based neural renderer enables it to accurately shade the object according to the desired light position, while properly harmonizing the object (and its shadow) within the target background image. Our method, SpotLight, leverages existing neural rendering approaches and achieves controllable relighting results with no additional training. Specifically, we demonstrate its use with two neural renderers from the recent literature. We show that SpotLight achieves superior object compositing results, both quantitatively and perceptually, as confirmed by a user study, outperforming existing diffusion-based models specifically designed for relighting.