ResearchTown：人類研究社區模擬器

ResearchTown: Simulator of Human Research Community

December 23, 2024

作者: Haofei Yu, Zhaochen Hong, Zirui Cheng, Kunlun Zhu, Keyang Xuan, Jinwei Yao, Tao Feng, Jiaxuan You

cs.AI

摘要

大型語言模型（LLMs）已在科學領域展現出卓越的潛力，然而一個基本問題仍然沒有答案：我們能用LLMs模擬人類研究社區嗎？解答這個問題可以加深我們對想法激發背後過程的理解，並激發對新科學見解的自動發現。在這項工作中，我們提出ResearchTown，一個用於研究社區模擬的多智能體框架。在這個框架內，人類研究社區被簡化並建模為一個智能體-數據圖，其中研究人員和論文分別被表示為智能體類型和數據類型節點，並根據他們的合作關係相連。我們還引入TextGNN，一個基於文本的推理框架，將各種研究活動（例如閱讀論文、寫作論文和寫作評論）建模為在智能體-數據圖上的統一消息傳遞過程的特殊形式。為了評估研究模擬的質量，我們提出ResearchBench，一個使用節點遮罩預測任務進行可擴展和客觀評估的基準。我們的實驗揭示了三個關鍵發現：（1）ResearchTown可以提供合作研究活動的逼真模擬，包括寫作論文和寫作評論；（2）ResearchTown可以保持對多個研究人員和多樣論文的穩健模擬；（3）ResearchTown可以產生跨學科研究想法，潛在地激發新的研究方向。

English

Large Language Models (LLMs) have demonstrated remarkable potential in scientific domains, yet a fundamental question remains unanswered: Can we simulate human research communities with LLMs? Addressing this question can deepen our understanding of the processes behind idea brainstorming and inspire the automatic discovery of novel scientific insights. In this work, we propose ResearchTown, a multi-agent framework for research community simulation. Within this framework, the human research community is simplified and modeled as an agent-data graph, where researchers and papers are represented as agent-type and data-type nodes, respectively, and connected based on their collaboration relationships. We also introduce TextGNN, a text-based inference framework that models various research activities (e.g., paper reading, paper writing, and review writing) as special forms of a unified message-passing process on the agent-data graph. To evaluate the quality of the research simulation, we present ResearchBench, a benchmark that uses a node-masking prediction task for scalable and objective assessment based on similarity. Our experiments reveal three key findings: (1) ResearchTown can provide a realistic simulation of collaborative research activities, including paper writing and review writing; (2) ResearchTown can maintain robust simulation with multiple researchers and diverse papers; (3) ResearchTown can generate interdisciplinary research ideas that potentially inspire novel research directions.