基于图神经动力学建模的动态3D高斯跟踪

Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modeling

October 24, 2024

作者: Mingtong Zhang, Kaifeng Zhang, Yunzhu Li

cs.AI

摘要

机器人与物体互动的视频编码了关于物体动态的丰富信息。然而，现有的视频预测方法通常没有明确考虑视频中的三维信息，例如机器人的动作和物体的三维状态，从而限制了它们在现实世界的机器人应用中的使用。在这项工作中，我们介绍了一个框架，通过明确考虑机器人的动作轨迹及其对场景动态的影响，直接从多视角RGB视频中学习物体动态。我们利用三维高斯飞溅（3DGS）的三维高斯表示来训练一个基于粒子的动力学模型，使用图神经网络。该模型在从密集跟踪的三维高斯重建中下采样的稀疏控制粒子上运行。通过在离线机器人互动数据上学习神经动力学模型，我们的方法可以预测在不同初始配置和未见机器人动作下的物体运动。高斯的三维变换可以从控制粒子的运动中进行插值，实现预测未来物体状态并实现动作条件的视频预测。动力学模型还可应用于基于模型的规划框架，用于物体操作任务。我们在各种可变形材料上进行实验，包括绳索、衣物和填充动物，展示了我们的框架对建模复杂形状和动态的能力。我们的项目页面可在https://gs-dynamics.github.io找到。

English

Videos of robots interacting with objects encode rich information about the objects' dynamics. However, existing video prediction approaches typically do not explicitly account for the 3D information from videos, such as robot actions and objects' 3D states, limiting their use in real-world robotic applications. In this work, we introduce a framework to learn object dynamics directly from multi-view RGB videos by explicitly considering the robot's action trajectories and their effects on scene dynamics. We utilize the 3D Gaussian representation of 3D Gaussian Splatting (3DGS) to train a particle-based dynamics model using Graph Neural Networks. This model operates on sparse control particles downsampled from the densely tracked 3D Gaussian reconstructions. By learning the neural dynamics model on offline robot interaction data, our method can predict object motions under varying initial configurations and unseen robot actions. The 3D transformations of Gaussians can be interpolated from the motions of control particles, enabling the rendering of predicted future object states and achieving action-conditioned video prediction. The dynamics model can also be applied to model-based planning frameworks for object manipulation tasks. We conduct experiments on various kinds of deformable materials, including ropes, clothes, and stuffed animals, demonstrating our framework's ability to model complex shapes and dynamics. Our project page is available at https://gs-dynamics.github.io.