SplatFields：用於稀疏3D和4D重建的神經高斯Splat

SplatFields: Neural Gaussian Splats for Sparse 3D and 4D Reconstruction

September 17, 2024

作者: Marko Mihajlovic, Sergey Prokudin, Siyu Tang, Robert Maier, Federica Bogo, Tony Tung, Edmond Boyer

cs.AI

摘要

從多視角圖像中對三維靜態場景和四維動態事件進行數字化一直是計算機視覺和圖形學中的一個挑戰。最近，三維高斯飛濺（3DGS）已經成為一種實用且可擴展的重建方法，因其出色的重建質量、實時渲染能力以及與廣泛使用的可視化工具兼容而受到歡迎。然而，該方法需要大量的輸入視角來實現高質量場景重建，這導致了一個重要的實際瓶頸。在捕捉動態場景時，這個挑戰尤為嚴重，因為部署大量攝像機陣列可能成本過高。在這項工作中，我們確定了高斯飛濺技術在稀疏重建環境中表現不佳的一個因素，即飛濺特徵缺乏空間自相關性。為了解決這個問題，我們提出了一種優化策略，通過將其建模為相應的隱式神經場的輸出，有效地規範了飛濺特徵。這將在各種情況下一致提升重建質量。我們的方法有效處理靜態和動態情況，通過在不同設置和場景複雜性下的廣泛測試加以證明。

English

Digitizing 3D static scenes and 4D dynamic events from multi-view images has long been a challenge in computer vision and graphics. Recently, 3D Gaussian Splatting (3DGS) has emerged as a practical and scalable reconstruction method, gaining popularity due to its impressive reconstruction quality, real-time rendering capabilities, and compatibility with widely used visualization tools. However, the method requires a substantial number of input views to achieve high-quality scene reconstruction, introducing a significant practical bottleneck. This challenge is especially severe in capturing dynamic scenes, where deploying an extensive camera array can be prohibitively costly. In this work, we identify the lack of spatial autocorrelation of splat features as one of the factors contributing to the suboptimal performance of the 3DGS technique in sparse reconstruction settings. To address the issue, we propose an optimization strategy that effectively regularizes splat features by modeling them as the outputs of a corresponding implicit neural field. This results in a consistent enhancement of reconstruction quality across various scenarios. Our approach effectively handles static and dynamic cases, as demonstrated by extensive testing across different setups and scene complexities.