基於圖神經動力建模的動態3D高斯追蹤

Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modeling

October 24, 2024

作者: Mingtong Zhang, Kaifeng Zhang, Yunzhu Li

cs.AI

摘要

機器人與物體互動的影片包含豐富的物體動態資訊。然而，現有的影片預測方法通常沒有明確考慮來自影片的三維資訊，例如機器人的動作和物體的三維狀態，這限制了它們在現實世界機器人應用中的使用。在這項研究中，我們提出了一個框架，通過明確考慮機器人的動作軌跡及其對場景動態的影響，從多視角RGB影片中直接學習物體動態。我們利用三維高斯擴散（3DGS）的三維高斯表示來訓練一個基於粒子的動態模型，並使用圖神經網絡。該模型在從密集追踪的三維高斯重建中下採樣的稀疏控制粒子上運行。通過在離線機器人互動數據上學習神經動態模型，我們的方法可以預測不同初始配置和未見過的機器人動作下的物體運動。高斯的三維轉換可以從控制粒子的運動進行插值，實現預測未來物體狀態並實現動作條件下的影片預測。動態模型還可應用於基於模型的規劃框架，用於物體操作任務。我們在各種可變形材料上進行實驗，包括繩子、衣服和填充動物，展示了我們的框架對建模複雜形狀和動態的能力。我們的專案頁面位於https://gs-dynamics.github.io。

English

Videos of robots interacting with objects encode rich information about the objects' dynamics. However, existing video prediction approaches typically do not explicitly account for the 3D information from videos, such as robot actions and objects' 3D states, limiting their use in real-world robotic applications. In this work, we introduce a framework to learn object dynamics directly from multi-view RGB videos by explicitly considering the robot's action trajectories and their effects on scene dynamics. We utilize the 3D Gaussian representation of 3D Gaussian Splatting (3DGS) to train a particle-based dynamics model using Graph Neural Networks. This model operates on sparse control particles downsampled from the densely tracked 3D Gaussian reconstructions. By learning the neural dynamics model on offline robot interaction data, our method can predict object motions under varying initial configurations and unseen robot actions. The 3D transformations of Gaussians can be interpolated from the motions of control particles, enabling the rendering of predicted future object states and achieving action-conditioned video prediction. The dynamics model can also be applied to model-based planning frameworks for object manipulation tasks. We conduct experiments on various kinds of deformable materials, including ropes, clothes, and stuffed animals, demonstrating our framework's ability to model complex shapes and dynamics. Our project page is available at https://gs-dynamics.github.io.