使用多智能体强化学习训练社交推理语言模型。

Training Language Models for Social Deduction with Multi-Agent Reinforcement Learning

February 9, 2025

作者: Bidipta Sarkar, Warren Xia, C. Karen Liu, Dorsa Sadigh

cs.AI

摘要

在多智能体环境中，使用自然语言进行交流是一种强大的工具，因为它使独立智能体能够在部分可观察的情境中共享信息，并允许与人类进行零-shot协调。然而，大多数先前的研究存在局限性，因为它们要么依赖于大量人类示范进行训练，要么缺乏生成自然且有用的交流策略的能力。在这项工作中，我们训练语言模型在自然语言中就其环境展开富有成效的讨论，而无需任何人类示范。我们将交流问题分解为倾听和表达两个部分。我们的关键思想是利用智能体的目标，将预测有关世界的有用信息作为密集奖励信号，以引导交流。具体来说，我们通过训练模型根据讨论内容预测环境信息来提高模型的倾听能力，并通过多智能体强化学习同时提高模型的表达能力，通过奖励消息的影响他智能体来实现。为了研究在复杂社交环境中交流的作用和必要性，我们研究了一款基于《谁是卧底》的具身社交推理游戏，其中需要回答的关键问题是对手卧底的身份。我们分析了由于我们的技术而产生的新行为，如指控嫌疑人和提供证据，并发现这种方法促进了充分的讨论，使胜率翻倍，相较于标准强化学习。我们在https://socialdeductionllm.github.io/发布了我们的代码和模型。

English

Communicating in natural language is a powerful tool in multi-agent settings, as it enables independent agents to share information in partially observable settings and allows zero-shot coordination with humans. However, most prior works are limited as they either rely on training with large amounts of human demonstrations or lack the ability to generate natural and useful communication strategies. In this work, we train language models to have productive discussions about their environment in natural language without any human demonstrations. We decompose the communication problem into listening and speaking. Our key idea is to leverage the agent's goal to predict useful information about the world as a dense reward signal that guides communication. Specifically, we improve a model's listening skills by training them to predict information about the environment based on discussions, and we simultaneously improve a model's speaking skills with multi-agent reinforcement learning by rewarding messages based on their influence on other agents. To investigate the role and necessity of communication in complex social settings, we study an embodied social deduction game based on Among Us, where the key question to answer is the identity of an adversarial imposter. We analyze emergent behaviors due to our technique, such as accusing suspects and providing evidence, and find that it enables strong discussions, doubling the win rates compared to standard RL. We release our code and models at https://socialdeductionllm.github.io/