LongBench v2：朝着对现实长文本多任务进行更深入理解和推理的方向发展

摘要

本文介绍了LongBench v2，这是一个旨在评估大型语言模型（LLMs）处理需要深度理解和推理跨现实世界多任务长上下文问题的基准测试。LongBench v2 包含503个具有挑战性的多项选择题，涵盖了从8k到2M字的上下文，涵盖了六个主要任务类别：单文档问答、多文档问答、长上下文学习、长对话历史理解、代码仓库理解和长结构化数据理解。为确保广度和实用性，我们从近100位受过良好教育且具有多样化专业背景的个人那里收集数据。我们采用自动化和手动审核流程来保持高质量和难度，结果表明在15分钟的时间限制下，人类专家仅能达到53.7%的准确率。我们的评估显示，当直接回答问题时，表现最佳的模型仅能达到50.1%的准确率。相比之下，包含更长推理的o1-preview模型达到了57.7%，超过人类基准4%。这些结果突显了增强推理能力和扩展推理时间计算的重要性，以解决LongBench v2 中的长上下文挑战。该项目网址为https://longbench2.github.io。

English

This paper introduces LongBench v2, a benchmark designed to assess the ability of LLMs to handle long-context problems requiring deep understanding and reasoning across real-world multitasks. LongBench v2 consists of 503 challenging multiple-choice questions, with contexts ranging from 8k to 2M words, across six major task categories: single-document QA, multi-document QA, long in-context learning, long-dialogue history understanding, code repository understanding, and long structured data understanding. To ensure the breadth and the practicality, we collect data from nearly 100 highly educated individuals with diverse professional backgrounds. We employ both automated and manual review processes to maintain high quality and difficulty, resulting in human experts achieving only 53.7% accuracy under a 15-minute time constraint. Our evaluation reveals that the best-performing model, when directly answers the questions, achieves only 50.1% accuracy. In contrast, the o1-preview model, which includes longer reasoning, achieves 57.7%, surpassing the human baseline by 4%. These results highlight the importance of enhanced reasoning ability and scaling inference-time compute to tackle the long-context challenges in LongBench v2. The project is available at https://longbench2.github.io.

LongBench v2：朝着对现实长文本多任务进行更深入理解和推理的方向发展

LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks

摘要

Summary

Support