多模态奖励基准：面向视觉语言模型奖励机制的综合评估

Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models

February 20, 2025

作者: Michihiro Yasunaga, Luke Zettlemoyer, Marjan Ghazvininejad

cs.AI

摘要

奖励模型在训练视觉-语言模型（VLMs）中扮演着关键角色，通过评估输出质量来实现与人类偏好的对齐。尽管其重要性不言而喻，研究界仍缺乏全面的开放基准来评估VLMs中的多模态奖励模型。为填补这一空白，我们推出了Multimodal RewardBench，这是一个专家标注的基准，涵盖六大领域：通用正确性、偏好、知识、推理、安全性和视觉问答。我们的数据集包含从多种VLMs中收集的5,211个标注的（提示、优选响应、拒绝响应）三元组。在评估一系列VLM评判者时，我们发现即使表现最佳的模型，如Gemini 1.5 Pro和Claude 3.5 Sonnet，整体准确率也仅为72%。值得注意的是，大多数模型在推理和安全性领域表现欠佳。这些发现表明，Multimodal RewardBench为跨多个领域推进奖励模型的发展提供了一个具有挑战性的测试平台。我们已在https://github.com/facebookresearch/multimodal_rewardbench上发布了该基准。

English

Reward models play an essential role in training vision-language models (VLMs) by assessing output quality to enable aligning with human preferences. Despite their importance, the research community lacks comprehensive open benchmarks for evaluating multimodal reward models in VLMs. To address this gap, we introduce Multimodal RewardBench, an expert-annotated benchmark covering six domains: general correctness, preference, knowledge, reasoning, safety, and visual question-answering. Our dataset comprises 5,211 annotated (prompt, chosen response, rejected response) triplets collected from various VLMs. In evaluating a range of VLM judges, we find that even the top-performing models, Gemini 1.5 Pro and Claude 3.5 Sonnet, achieve only 72% overall accuracy. Notably, most models struggle in the reasoning and safety domains. These findings suggest that Multimodal RewardBench offers a challenging testbed for advancing reward model development across multiple domains. We release the benchmark at https://github.com/facebookresearch/multimodal_rewardbench.