Emma-X：一個具體的多模態行動模型，具有基於連貫思維和前瞻空間推理的基礎链。

Emma-X: An Embodied Multimodal Action Model with Grounded Chain of Thought and Look-ahead Spatial Reasoning

December 16, 2024

作者: Qi Sun, Pengfei Hong, Tej Deep Pala, Vernon Toh, U-Xuan Tan, Deepanway Ghosal, Soujanya Poria

cs.AI

摘要

傳統的基於強化學習的機器人控制方法通常是特定任務的，無法泛化到不同環境或未見過的物體和指令。視覺語言模型（VLMs）展示了強大的場景理解和規劃能力，但缺乏生成針對特定機器人實體的可操作策略的能力。為了解決這個問題，出現了視覺-語言-動作（VLA）模型，但它們在長時間跨度的空間推理和基於任務的規劃方面面臨挑戰。在這項工作中，我們提出了具有基於鏈式思維和前瞻空間推理的具體多模態行動模型，Emma-X。Emma-X利用我們基於BridgeV2構建的階層實體數據集，其中包含60,000個機器人操作軌跡，自動註釋了基於任務的推理和空間引導。此外，我們引入了一種基於夾爪狀態和運動軌跡的軌跡分割策略，可以幫助減輕在基於地面子任務推理生成中的幻覺。實驗結果表明，Emma-X在需要空間推理的真實世界機器人任務中，相對競爭基線實現了卓越的性能。

English

Traditional reinforcement learning-based robotic control methods are often task-specific and fail to generalize across diverse environments or unseen objects and instructions. Visual Language Models (VLMs) demonstrate strong scene understanding and planning capabilities but lack the ability to generate actionable policies tailored to specific robotic embodiments. To address this, Visual-Language-Action (VLA) models have emerged, yet they face challenges in long-horizon spatial reasoning and grounded task planning. In this work, we propose the Embodied Multimodal Action Model with Grounded Chain of Thought and Look-ahead Spatial Reasoning, Emma-X. Emma-X leverages our constructed hierarchical embodiment dataset based on BridgeV2, containing 60,000 robot manipulation trajectories auto-annotated with grounded task reasoning and spatial guidance. Additionally, we introduce a trajectory segmentation strategy based on gripper states and motion trajectories, which can help mitigate hallucination in grounding subtask reasoning generation. Experimental results demonstrate that Emma-X achieves superior performance over competitive baselines, particularly in real-world robotic tasks requiring spatial reasoning.