BALROG：在游戏中对代理式LLM和VLM推理进行基准测试

BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games

November 20, 2024

作者: Davide Paglieri, Bartłomiej Cupiał, Samuel Coward, Ulyana Piterbarg, Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuciński, Lerrel Pinto, Rob Fergus, Jakob Nicolaus Foerster, Jack Parker-Holder, Tim Rocktäschel

cs.AI

摘要

大型语言模型（LLMs）和视觉语言模型（VLMs）具有广泛的知识并展现出有前途的推理能力；然而，它们仍然在复杂、动态环境中表现不佳。现实世界的任务需要处理错综复杂的互动、高级空间推理、长期规划以及持续探索新策略等领域，而我们缺乏有效的方法来全面评估这些能力。为了弥补这一空白，我们引入了BALROG，一个旨在通过一系列具有挑战性的游戏评估LLMs和VLMs代理能力的新基准。我们的基准包含一系列现有的强化学习环境，难度各异，包括一些能够在几秒内由非专家人员解决的任务，到可能需要数年才能掌握的极具挑战性的任务（例如NetHack学习环境）。我们设计了细粒度的指标来衡量性能，并对几种流行的开源和闭源LLMs和VLMs进行了广泛评估。我们的研究结果表明，当前模型在较简单的游戏中取得了部分成功，但在更具挑战性的任务中遇到了重大困难。值得注意的是，我们观察到在基于视觉的决策制定方面存在严重的不足，因为当环境的视觉表示提供时，模型的表现更差。我们将BALROG作为一个开放且用户友好的基准发布，以促进代理社区未来的研究和发展。

English

Large Language Models (LLMs) and Vision Language Models (VLMs) possess extensive knowledge and exhibit promising reasoning abilities; however, they still struggle to perform well in complex, dynamic environments. Real-world tasks require handling intricate interactions, advanced spatial reasoning, long-term planning, and continuous exploration of new strategies-areas in which we lack effective methodologies for comprehensively evaluating these capabilities. To address this gap, we introduce BALROG, a novel benchmark designed to assess the agentic capabilities of LLMs and VLMs through a diverse set of challenging games. Our benchmark incorporates a range of existing reinforcement learning environments with varying levels of difficulty, including tasks that are solvable by non-expert humans in seconds to extremely challenging ones that may take years to master (e.g., the NetHack Learning Environment). We devise fine-grained metrics to measure performance and conduct an extensive evaluation of several popular open-source and closed-source LLMs and VLMs. Our findings indicate that while current models achieve partial success in the easier games, they struggle significantly with more challenging tasks. Notably, we observe severe deficiencies in vision-based decision-making, as models perform worse when visual representations of the environments are provided. We release BALROG as an open and user-friendly benchmark to facilitate future research and development in the agentic community.