OmniManip：通过以物体为中心的交互基元作为空间约束，实现通用机器人操作

OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints

January 7, 2025

作者: Mingjie Pan, Jiyao Zhang, Tianshu Wu, Yinghao Zhao, Wenlong Gao, Hao Dong

cs.AI

摘要

在无结构环境中进行操纵的通用机器人系统的开发是一个重大挑战。虽然视觉语言模型（VLM）擅长高层次常识推理，但它们缺乏精细的三维空间理解，这是精确操纵任务所需的。在机器人数据集上微调VLM以创建视觉-语言-动作模型（VLA）是一个潜在的解决方案，但受到高数据收集成本和泛化问题的阻碍。为了解决这些挑战，我们提出了一种新颖的以物体为中心的表示，弥合了VLM高层次推理和操纵所需的低级精度之间的差距。我们的关键见解是，物体的规范空间，由其功能性可供性定义，提供了一种结构化和语义上有意义的描述交互基元（如点和方向）的方式。这些基元充当桥梁，将VLM的常识推理转化为可操作的三维空间约束。在这种背景下，我们引入了一个双闭环、开词汇的机器人操纵系统：一个用于通过基元重采样、交互渲染和VLM检查进行高层规划的闭环，另一个用于通过6D姿态跟踪进行低级执行。这种设计确保了强大的、实时的控制，而无需进行VLM微调。大量实验展示了在各种机器人操纵任务中强大的零样本泛化能力，突显了这种方法在自动化大规模模拟数据生成方面的潜力。

English

The development of general robotic systems capable of manipulating in unstructured environments is a significant challenge. While Vision-Language Models(VLM) excel in high-level commonsense reasoning, they lack the fine-grained 3D spatial understanding required for precise manipulation tasks. Fine-tuning VLM on robotic datasets to create Vision-Language-Action Models(VLA) is a potential solution, but it is hindered by high data collection costs and generalization issues. To address these challenges, we propose a novel object-centric representation that bridges the gap between VLM's high-level reasoning and the low-level precision required for manipulation. Our key insight is that an object's canonical space, defined by its functional affordances, provides a structured and semantically meaningful way to describe interaction primitives, such as points and directions. These primitives act as a bridge, translating VLM's commonsense reasoning into actionable 3D spatial constraints. In this context, we introduce a dual closed-loop, open-vocabulary robotic manipulation system: one loop for high-level planning through primitive resampling, interaction rendering and VLM checking, and another for low-level execution via 6D pose tracking. This design ensures robust, real-time control without requiring VLM fine-tuning. Extensive experiments demonstrate strong zero-shot generalization across diverse robotic manipulation tasks, highlighting the potential of this approach for automating large-scale simulation data generation.