反映現實：使擴散模型能夠產生忠實的鏡像反射

Reflecting Reality: Enabling Diffusion Models to Produce Faithful Mirror Reflections

September 23, 2024

作者: Ankit Dhiman, Manan Shah, Rishubh Parihar, Yash Bhalgat, Lokesh R Boregowda, R Venkatesh Babu

cs.AI

摘要

我們致力於使用基於擴散的生成模型來生成高度逼真和合理的鏡像反射問題。我們將這個問題定義為圖像修補任務，允許在生成過程中更多地控制鏡子的放置。為了實現這一點，我們創建了SynMirror，這是一個大規模的數據集，其中包含在鏡子前放置物體的各種合成場景。SynMirror包含約198K個樣本，從66K個獨特的3D物體渲染而來，還包括它們相應的深度圖、法線圖和實例分割遮罩，以捕捉場景的相關幾何特性。利用這個數據集，我們提出了一種新穎的深度條件修補方法，稱為MirrorFusion，它可以生成高質量、幾何一致且照片逼真的鏡像反射，只需提供輸入圖像和描述鏡子區域的遮罩。通過廣泛的定量和定性分析，MirrorFusion在SynMirror上表現優於最先進的方法。據我們所知，我們是第一個成功應用基於擴散的模型解決生成場景中物體的受控和忠實的鏡像反射問題的研究者。SynMirror和MirrorFusion為從業者和研究人員開闢了圖像編輯和擴增實境應用的新途徑。

English

We tackle the problem of generating highly realistic and plausible mirror reflections using diffusion-based generative models. We formulate this problem as an image inpainting task, allowing for more user control over the placement of mirrors during the generation process. To enable this, we create SynMirror, a large-scale dataset of diverse synthetic scenes with objects placed in front of mirrors. SynMirror contains around 198K samples rendered from 66K unique 3D objects, along with their associated depth maps, normal maps and instance-wise segmentation masks, to capture relevant geometric properties of the scene. Using this dataset, we propose a novel depth-conditioned inpainting method called MirrorFusion, which generates high-quality geometrically consistent and photo-realistic mirror reflections given an input image and a mask depicting the mirror region. MirrorFusion outperforms state-of-the-art methods on SynMirror, as demonstrated by extensive quantitative and qualitative analysis. To the best of our knowledge, we are the first to successfully tackle the challenging problem of generating controlled and faithful mirror reflections of an object in a scene using diffusion based models. SynMirror and MirrorFusion open up new avenues for image editing and augmented reality applications for practitioners and researchers alike.