高斯属性：将物理属性集成到具有LMMs的3D高斯函数中

GaussianProperty: Integrating Physical Properties to 3D Gaussians with LMMs

December 15, 2024

作者: Xinli Xu, Wenhang Ge, Dicong Qiu, ZhiFei Chen, Dongyu Yan, Zhuoyun Liu, Haoyu Zhao, Hanfeng Zhao, Shunsi Zhang, Junwei Liang, Ying-Cong Chen

cs.AI

摘要

在计算机视觉、图形学和机器人领域，估计视觉数据的物理特性是一项关键任务，支撑着增强现实、物理模拟和机器人抓取等应用。然而，由于物理特性估计中固有的歧义，这一领域仍未得到充分探索。为了解决这些挑战，我们引入了GaussianProperty，这是一个无需训练的框架，将材料的物理特性分配给3D高斯分布。具体来说，我们将SAM的分割能力与GPT-4V(ision)的识别能力相结合，为2D图像制定了一个全局-局部的物理特性推理模块。然后，我们使用投票策略将多视角2D图像中的物理特性投影到3D高斯分布中。我们证明，带有物理特性注释的3D高斯分布可以应用于基于物理的动态模拟和机器人抓取。对于基于物理的动态模拟，我们利用材料点法（MPM）进行逼真的动态模拟。对于机器人抓取，我们开发了一个抓取力预测策略，根据估计的物理特性来估计抓取物体所需的安全力范围。对材料分割、基于物理的动态模拟和机器人抓取进行的大量实验验证了我们提出方法的有效性，突显了其在从视觉数据中理解物理特性方面的关键作用。在线演示、代码、更多案例和带注释数据集可在https://Gaussian-Property.github.io{此 https URL}上找到。

English

Estimating physical properties for visual data is a crucial task in computer vision, graphics, and robotics, underpinning applications such as augmented reality, physical simulation, and robotic grasping. However, this area remains under-explored due to the inherent ambiguities in physical property estimation. To address these challenges, we introduce GaussianProperty, a training-free framework that assigns physical properties of materials to 3D Gaussians. Specifically, we integrate the segmentation capability of SAM with the recognition capability of GPT-4V(ision) to formulate a global-local physical property reasoning module for 2D images. Then we project the physical properties from multi-view 2D images to 3D Gaussians using a voting strategy. We demonstrate that 3D Gaussians with physical property annotations enable applications in physics-based dynamic simulation and robotic grasping. For physics-based dynamic simulation, we leverage the Material Point Method (MPM) for realistic dynamic simulation. For robot grasping, we develop a grasping force prediction strategy that estimates a safe force range required for object grasping based on the estimated physical properties. Extensive experiments on material segmentation, physics-based dynamic simulation, and robotic grasping validate the effectiveness of our proposed method, highlighting its crucial role in understanding physical properties from visual data. Online demo, code, more cases and annotated datasets are available on https://Gaussian-Property.github.io{this https URL}.