高斯性質：將物理特性整合到具有LMMs的3D高斯函數

GaussianProperty: Integrating Physical Properties to 3D Gaussians with LMMs

December 15, 2024

作者: Xinli Xu, Wenhang Ge, Dicong Qiu, ZhiFei Chen, Dongyu Yan, Zhuoyun Liu, Haoyu Zhao, Hanfeng Zhao, Shunsi Zhang, Junwei Liang, Ying-Cong Chen

cs.AI

摘要

在計算機視覺、圖形學和機器人技術中，估算視覺數據的物理特性是一項至關重要的任務，支撐著擴增實境、物理模擬和機器人抓取等應用。然而，由於物理特性估算中固有的歧義性，這一領域仍未得到充分探索。為應對這些挑戰，我們引入了 GaussianProperty，一個無需訓練的框架，將材料的物理特性分配給 3D 高斯分布。具體而言，我們將 SAM 的分割能力與 GPT-4V(ision) 的識別能力相結合，形成一個針對 2D 圖像的全局-局部物理特性推理模塊。然後，我們使用投票策略將多視角 2D 圖像中的物理特性投影到 3D 高斯分布中。我們展示了帶有物理特性標註的 3D 高斯分布使得應用於基於物理的動態模擬和機器人抓取成為可能。對於基於物理的動態模擬，我們利用材料點法（MPM）進行逼真的動態模擬。對於機器人抓取，我們制定了一種抓取力預測策略，根據估算的物理特性來估計對象抓取所需的安全力範圍。對材料分割、基於物理的動態模擬和機器人抓取進行的大量實驗驗證了我們提出方法的有效性，凸顯了它在從視覺數據中理解物理特性方面的關鍵作用。在 https://Gaussian-Property.github.io{此https URL} 上提供了在線演示、代碼、更多案例和標註數據集。

English

Estimating physical properties for visual data is a crucial task in computer vision, graphics, and robotics, underpinning applications such as augmented reality, physical simulation, and robotic grasping. However, this area remains under-explored due to the inherent ambiguities in physical property estimation. To address these challenges, we introduce GaussianProperty, a training-free framework that assigns physical properties of materials to 3D Gaussians. Specifically, we integrate the segmentation capability of SAM with the recognition capability of GPT-4V(ision) to formulate a global-local physical property reasoning module for 2D images. Then we project the physical properties from multi-view 2D images to 3D Gaussians using a voting strategy. We demonstrate that 3D Gaussians with physical property annotations enable applications in physics-based dynamic simulation and robotic grasping. For physics-based dynamic simulation, we leverage the Material Point Method (MPM) for realistic dynamic simulation. For robot grasping, we develop a grasping force prediction strategy that estimates a safe force range required for object grasping based on the estimated physical properties. Extensive experiments on material segmentation, physics-based dynamic simulation, and robotic grasping validate the effectiveness of our proposed method, highlighting its crucial role in understanding physical properties from visual data. Online demo, code, more cases and annotated datasets are available on https://Gaussian-Property.github.io{this https URL}.