研究城：人类研究社区模拟器

ResearchTown: Simulator of Human Research Community

December 23, 2024

作者: Haofei Yu, Zhaochen Hong, Zirui Cheng, Kunlun Zhu, Keyang Xuan, Jinwei Yao, Tao Feng, Jiaxuan You

cs.AI

摘要

大型语言模型（LLMs）在科学领域展现出了显著的潜力，然而一个基本问题仍然没有得到解答：我们能用LLMs模拟人类研究社区吗？解决这个问题可以加深我们对头脑风暴背后的过程的理解，并激发自动发现新科学见解的灵感。在这项工作中，我们提出了ResearchTown，一个用于研究社区模拟的多智能体框架。在这个框架内，人类研究社区被简化并建模为一个智能体-数据图，其中研究人员和论文分别表示为智能体类型和数据类型节点，并根据他们的合作关系连接。我们还引入了TextGNN，一个基于文本的推理框架，将各种研究活动（如阅读论文、撰写论文和审阅写作）建模为在智能体-数据图上的统一消息传递过程的特殊形式。为了评估研究模拟的质量，我们提出了ResearchBench，一个使用节点掩码预测任务进行可伸缩和客观评估的基准。我们的实验揭示了三个关键发现：（1）ResearchTown可以提供对合作研究活动的逼真模拟，包括论文撰写和审阅写作；（2）ResearchTown可以保持与多个研究人员和多样化论文的稳健模拟；（3）ResearchTown可以产生激发新研究方向的跨学科研究思路。

English

Large Language Models (LLMs) have demonstrated remarkable potential in scientific domains, yet a fundamental question remains unanswered: Can we simulate human research communities with LLMs? Addressing this question can deepen our understanding of the processes behind idea brainstorming and inspire the automatic discovery of novel scientific insights. In this work, we propose ResearchTown, a multi-agent framework for research community simulation. Within this framework, the human research community is simplified and modeled as an agent-data graph, where researchers and papers are represented as agent-type and data-type nodes, respectively, and connected based on their collaboration relationships. We also introduce TextGNN, a text-based inference framework that models various research activities (e.g., paper reading, paper writing, and review writing) as special forms of a unified message-passing process on the agent-data graph. To evaluate the quality of the research simulation, we present ResearchBench, a benchmark that uses a node-masking prediction task for scalable and objective assessment based on similarity. Our experiments reveal three key findings: (1) ResearchTown can provide a realistic simulation of collaborative research activities, including paper writing and review writing; (2) ResearchTown can maintain robust simulation with multiple researchers and diverse papers; (3) ResearchTown can generate interdisciplinary research ideas that potentially inspire novel research directions.