MVGS：多視角調節高斯塗抹用於新視角合成

MVGS: Multi-view-regulated Gaussian Splatting for Novel View Synthesis

October 2, 2024

作者: Xiaobiao Du, Yida Wang, Xin Yu

cs.AI

摘要

最近在體積渲染方面的研究，例如 NeRF 和 3D 高斯光斑（3DGS），在學習到的隱式神經輻射場或 3D 高斯函數的幫助下，顯著提高了渲染質量和效率。在明確表示的基礎上，普通的 3DGS 及其變體通過在訓練期間每次迭代優化參數模型以單視圖監督的方式實現了實時效率，這種方法源自 NeRF。因此，某些視圖被過度擬合，導致新視圖合成的外觀不理想且 3D 幾何形狀不精確。為了解決上述問題，我們提出了一種新的 3DGS 優化方法，具有四個關鍵的新貢獻：1）我們將傳統的單視圖訓練範式轉換為多視圖訓練策略。通過我們提出的多視圖調節，進一步優化 3D 高斯函數的屬性，避免過度擬合某些訓練視圖。作為一種通用解決方案，我們在各種情境和不同高斯變體中提高了整體準確性。2）受到額外視圖帶來的好處的啟發，我們進一步提出了一種交叉內在引導方案，引導進行關於不同分辨率的從粗到細的訓練程序。3）在我們的多視圖調節訓練基礎上，我們進一步提出了一種交叉射線密集化策略，從選定的視圖中在射線交叉區域中加入更多高斯核。4）通過進一步研究密集化策略，我們發現當某些視圖差異明顯時，密集化效果應當增強。作為解決方案，我們提出了一種新穎的多視圖增強密集化策略，鼓勵 3D 高斯函數根據需要增加密集化，從而提高重建準確性。

English

Recent works in volume rendering, e.g. NeRF and 3D Gaussian Splatting (3DGS), significantly advance the rendering quality and efficiency with the help of the learned implicit neural radiance field or 3D Gaussians. Rendering on top of an explicit representation, the vanilla 3DGS and its variants deliver real-time efficiency by optimizing the parametric model with single-view supervision per iteration during training which is adopted from NeRF. Consequently, certain views are overfitted, leading to unsatisfying appearance in novel-view synthesis and imprecise 3D geometries. To solve aforementioned problems, we propose a new 3DGS optimization method embodying four key novel contributions: 1) We transform the conventional single-view training paradigm into a multi-view training strategy. With our proposed multi-view regulation, 3D Gaussian attributes are further optimized without overfitting certain training views. As a general solution, we improve the overall accuracy in a variety of scenarios and different Gaussian variants. 2) Inspired by the benefit introduced by additional views, we further propose a cross-intrinsic guidance scheme, leading to a coarse-to-fine training procedure concerning different resolutions. 3) Built on top of our multi-view regulated training, we further propose a cross-ray densification strategy, densifying more Gaussian kernels in the ray-intersect regions from a selection of views. 4) By further investigating the densification strategy, we found that the effect of densification should be enhanced when certain views are distinct dramatically. As a solution, we propose a novel multi-view augmented densification strategy, where 3D Gaussians are encouraged to get densified to a sufficient number accordingly, resulting in improved reconstruction accuracy.