Magma：多模态AI智能体的基础模型

Magma: A Foundation Model for Multimodal AI Agents

February 18, 2025

作者: Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, Yuquan Deng, Lars Liden, Jianfeng Gao

cs.AI

摘要

我们推出Magma，一个面向数字与物理世界的多模态AI代理任务的基础模型。Magma在视觉-语言（VL）模型基础上实现了重大扩展，不仅保留了后者对视觉语言的理解能力（言语智能），还赋予了在视觉-空间世界中规划与行动（时空智能）的能力，能够完成从用户界面导航到机器人操控等一系列代理任务。为了赋予这些代理能力，Magma在大量异构数据集上进行了预训练，涵盖图像、视频乃至机器人数据，其中图像中的可操作视觉对象（如GUI中的可点击按钮）通过标记集（SoM）进行标注以实现动作定位，而视频中物体运动（如人手或机械臂的轨迹）则通过轨迹标记（ToM）进行标注以支持动作规划。大量实验表明，SoM与ToM达到了极佳的协同效应，促进了Magma模型时空智能的习得，这对于图1所示的一系列任务至关重要。特别地，Magma在用户界面导航和机器人操控任务上创造了新的最佳成绩，超越了以往专门针对这些任务设计的模型。在图像和视频相关的多模态任务上，Magma同样表现优异，与那些在更大规模数据集上训练的大型多模态模型相比毫不逊色。我们公开了模型及代码以促进可复现性，详情请访问https://microsoft.github.io/Magma。

English

We present Magma, a foundation model that serves multimodal AI agentic tasks in both the digital and physical worlds. Magma is a significant extension of vision-language (VL) models in that it not only retains the VL understanding ability (verbal intelligence) of the latter, but is also equipped with the ability to plan and act in the visual-spatial world (spatial-temporal intelligence) and complete agentic tasks ranging from UI navigation to robot manipulation. To endow the agentic capabilities, Magma is pretrained on large amounts of heterogeneous datasets spanning from images, videos to robotics data, where the actionable visual objects (e.g., clickable buttons in GUI) in images are labeled by Set-of-Mark (SoM) for action grounding, and the object movements (e.g., the trace of human hands or robotic arms) in videos are labeled by Trace-of-Mark (ToM) for action planning. Extensive experiments show that SoM and ToM reach great synergy and facilitate the acquisition of spatial-temporal intelligence for our Magma model, which is fundamental to a wide range of tasks as shown in Fig.1. In particular, Magma creates new state-of-the-art results on UI navigation and robotic manipulation tasks, outperforming previous models that are specifically tailored to these tasks. On image and video-related multimodal tasks, Magma also compares favorably to popular large multimodal models that are trained on much larger datasets. We make our model and code public for reproducibility at https://microsoft.github.io/Magma.