揭示强化学习智能体记忆复杂性的方法：一种分类和评估方法

Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluation

December 9, 2024

作者: Egor Cherepanov, Nikita Kachaev, Artem Zholus, Alexey K. Kovalev, Aleksandr I. Panov

cs.AI

摘要

将记忆融入智能体对于强化学习（RL）领域内众多任务至关重要。特别是，对于需要利用过去信息、适应新环境以及提高样本效率的任务，记忆至关重要。然而，“记忆”一词涵盖了广泛的概念，再加上缺乏统一的验证智能体记忆的方法，导致对智能体记忆能力的错误判断，并阻止与其他增强记忆智能体客观比较。本文旨在通过提供灵感来源于认知科学的智能体记忆类型的实用精确定义，如长期记忆与短期记忆、陈述性记忆与程序性记忆，来简化RL中记忆的概念。利用这些定义，我们对智能体记忆的不同类别进行分类，提出了一个评估RL智能体记忆能力的强大实验方法，并标准化评估。此外，我们通过对不同RL智能体进行实验，以及违反该方法论的后果，从经验上证明了遵循所提方法论在评估不同类型智能体记忆时的重要性。

English

The incorporation of memory into agents is essential for numerous tasks within the domain of Reinforcement Learning (RL). In particular, memory is paramount for tasks that require the utilization of past information, adaptation to novel environments, and improved sample efficiency. However, the term ``memory'' encompasses a wide range of concepts, which, coupled with the lack of a unified methodology for validating an agent's memory, leads to erroneous judgments about agents' memory capabilities and prevents objective comparison with other memory-enhanced agents. This paper aims to streamline the concept of memory in RL by providing practical precise definitions of agent memory types, such as long-term versus short-term memory and declarative versus procedural memory, inspired by cognitive science. Using these definitions, we categorize different classes of agent memory, propose a robust experimental methodology for evaluating the memory capabilities of RL agents, and standardize evaluations. Furthermore, we empirically demonstrate the importance of adhering to the proposed methodology when evaluating different types of agent memory by conducting experiments with different RL agents and what its violation leads to.