AAAR-1.0：評估人工智慧協助研究的潛力

摘要

許多研究已評估人工智慧系統的能力，特別是大型語言模型（LLMs），在促進日常任務方面的表現，如電子郵件撰寫、問答和創意內容生成。然而，研究人員在利用LLMs進行自身工作時面臨獨特的挑戰和機遇，例如構思研究想法、設計實驗以及撰寫或審查論文。在本研究中，我們介紹AAAR-1.0，一個旨在評估LLM在三個基本、專業密集的研究任務中的表現的基準數據集：（i）EquationInference，根據論文提交中的上下文信息評估方程式的正確性；（ii）ExperimentDesign，設計實驗以驗證研究想法和解決方案；（iii）PaperWeakness，識別論文提交中的弱點；以及（iv）REVIEWCRITIQUE，識別人類評論中每個部分是否不足。AAAR-1.0在兩個關鍵方面與先前的基準數據集不同：首先，它明確針對研究，任務需要深入的領域專業知識；其次，它針對研究人員，反映研究人員每天主要從事的活動。對於開源和專有LLMs的評估顯示了它們在進行複雜研究任務方面的潛力以及限制。我們將繼續對AAAR-1.0進行更新。

English

Numerous studies have assessed the proficiency of AI systems, particularly large language models (LLMs), in facilitating everyday tasks such as email writing, question answering, and creative content generation. However, researchers face unique challenges and opportunities in leveraging LLMs for their own work, such as brainstorming research ideas, designing experiments, and writing or reviewing papers. In this study, we introduce AAAR-1.0, a benchmark dataset designed to evaluate LLM performance in three fundamental, expertise-intensive research tasks: (i) EquationInference, assessing the correctness of equations based on the contextual information in paper submissions; (ii) ExperimentDesign, designing experiments to validate research ideas and solutions; (iii) PaperWeakness, identifying weaknesses in paper submissions; and (iv) REVIEWCRITIQUE, identifying each segment in human reviews is deficient or not. AAAR-1.0 differs from prior benchmarks in two key ways: first, it is explicitly research-oriented, with tasks requiring deep domain expertise; second, it is researcher-oriented, mirroring the primary activities that researchers engage in on a daily basis. An evaluation of both open-source and proprietary LLMs reveals their potential as well as limitations in conducting sophisticated research tasks. We will keep iterating AAAR-1.0 to new versions.

AAAR-1.0：評估人工智慧協助研究的潛力

AAAR-1.0: Assessing AI's Potential to Assist Research

摘要

Summary

Support

Support