3DGS-Enhancer：透過與視角一致的2D擴散先驗來增強無界3D高斯潑灑

3DGS-Enhancer: Enhancing Unbounded 3D Gaussian Splatting with View-consistent 2D Diffusion Priors

October 21, 2024

作者: Xi Liu, Chaoyi Zhou, Siyu Huang

cs.AI

摘要

新視角合成旨在從多個輸入圖像或視頻中生成場景的新視角，最近的進展如3D高斯飛灑（3DGS）在具有高效管道的情況下實現了產生逼真渲染的顯著成功。然而，在具有挑戰性設置下生成高質量的新視角，例如稀疏輸入視角，由於欠採樣區域中信息不足，通常導致明顯的瑕疵。本文提出了3DGS-Enhancer，一種用於增強3DGS表示質量的新型管道。我們利用2D視頻擴散先驗來解決具有挑戰性的3D視角一致性問題，將其重新制定為實現視頻生成過程中的時間一致性。3DGS-Enhancer恢復了渲染的新視角的視角一致潛在特徵，並通過空間-時間解碼器將其與輸入視角集成。增強的視圖然後用於微調初始3DGS模型，顯著提高了其渲染性能。對無邊界場景的大規模數據集進行了大量實驗，證明了3DGS-Enhancer相對於最先進方法具有優越的重建性能和高保真渲染結果。項目網頁為https://xiliu8006.github.io/3DGS-Enhancer-project。

English

Novel-view synthesis aims to generate novel views of a scene from multiple input images or videos, and recent advancements like 3D Gaussian splatting (3DGS) have achieved notable success in producing photorealistic renderings with efficient pipelines. However, generating high-quality novel views under challenging settings, such as sparse input views, remains difficult due to insufficient information in under-sampled areas, often resulting in noticeable artifacts. This paper presents 3DGS-Enhancer, a novel pipeline for enhancing the representation quality of 3DGS representations. We leverage 2D video diffusion priors to address the challenging 3D view consistency problem, reformulating it as achieving temporal consistency within a video generation process. 3DGS-Enhancer restores view-consistent latent features of rendered novel views and integrates them with the input views through a spatial-temporal decoder. The enhanced views are then used to fine-tune the initial 3DGS model, significantly improving its rendering performance. Extensive experiments on large-scale datasets of unbounded scenes demonstrate that 3DGS-Enhancer yields superior reconstruction performance and high-fidelity rendering results compared to state-of-the-art methods. The project webpage is https://xiliu8006.github.io/3DGS-Enhancer-project .