PhysBench：为物理世界理解基础上的视觉-语言模型进行基准测试和增强

PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding

January 27, 2025

作者: Wei Chow, Jiageng Mao, Boyi Li, Daniel Seita, Vitor Guizilini, Yue Wang

cs.AI

摘要

在具身体性的人工智能研究中，理解现实世界是一个基本挑战，对于使代理能够执行复杂任务并在真实环境中安全运行至关重要。虽然视觉语言模型（VLMs）在推理和任务规划方面表现出极大的潜力，但它们理解物理现象的能力仍然非常有限。为了弥补这一差距，我们引入了PhysBench，这是一个全面的基准测试，旨在评估VLMs在各种任务中对物理世界理解能力。 PhysBench包含10,002个交织的视频-图像-文本数据条目，分为四个主要领域：物理对象属性、物理对象关系、物理场景理解和基于物理的动态，进一步细分为19个子类和8个不同的能力维度。我们进行了大量实验，针对75个代表性的VLMs进行了实验，结果显示，尽管这些模型在常识推理方面表现出色，但它们在理解物理世界方面存在困难，这可能是因为它们的训练数据中缺乏物理知识并且缺乏嵌入的物理先验知识。为了解决这一不足，我们引入了PhysAgent，这是一个结合了VLMs的泛化优势和视觉模型专业知识的新框架，显著增强了VLMs在各种任务中对物理理解，包括对GPT-4o的18.4\%改进。此外，我们的结果表明，增强VLMs对物理世界理解能力可以帮助像MOKA这样的具身体性代理。我们相信，PhysBench和PhysAgent提供了宝贵的见解，并有助于弥合VLMs与物理世界理解之间的差距。

English

Understanding the physical world is a fundamental challenge in embodied AI, critical for enabling agents to perform complex tasks and operate safely in real-world environments. While Vision-Language Models (VLMs) have shown great promise in reasoning and task planning for embodied agents, their ability to comprehend physical phenomena remains extremely limited. To close this gap, we introduce PhysBench, a comprehensive benchmark designed to evaluate VLMs' physical world understanding capability across a diverse set of tasks. PhysBench contains 10,002 entries of interleaved video-image-text data, categorized into four major domains: physical object properties, physical object relationships, physical scene understanding, and physics-based dynamics, further divided into 19 subclasses and 8 distinct capability dimensions. Our extensive experiments, conducted on 75 representative VLMs, reveal that while these models excel in common-sense reasoning, they struggle with understanding the physical world -- likely due to the absence of physical knowledge in their training data and the lack of embedded physical priors. To tackle the shortfall, we introduce PhysAgent, a novel framework that combines the generalization strengths of VLMs with the specialized expertise of vision models, significantly enhancing VLMs' physical understanding across a variety of tasks, including an 18.4\% improvement on GPT-4o. Furthermore, our results demonstrate that enhancing VLMs' physical world understanding capabilities can help embodied agents such as MOKA. We believe that PhysBench and PhysAgent offer valuable insights and contribute to bridging the gap between VLMs and physical world understanding.