揭開強化學習智能體記憶複雜性的方法：分類與評估

摘要

將記憶融入代理人對於強化學習（RL）領域內眾多任務至關重要。特別是，記憶對於需要利用過去資訊、適應新環境和提高樣本效率的任務至關重要。然而，“記憶”一詞涵蓋了廣泛的概念，再加上缺乏統一的方法來驗證代理人記憶，導致對代理人記憶能力的錯誤判斷，並阻礙與其他增強記憶代理人客觀比較。本文旨在通過提供實用的準確定義來精簡RL中的記憶概念，例如長期記憶與短期記憶、陳述性記憶與程序性記憶等，靈感來自認知科學。利用這些定義，我們將代理人記憶的不同類別進行分類，提出了一種強化學習代理人記憶能力評估的穩健實驗方法，並標準化評估。此外，我們通過對不同RL代理人進行實驗來實證遵循所提出方法的重要性，以及其違反將導致的後果。

English

The incorporation of memory into agents is essential for numerous tasks within the domain of Reinforcement Learning (RL). In particular, memory is paramount for tasks that require the utilization of past information, adaptation to novel environments, and improved sample efficiency. However, the term ``memory'' encompasses a wide range of concepts, which, coupled with the lack of a unified methodology for validating an agent's memory, leads to erroneous judgments about agents' memory capabilities and prevents objective comparison with other memory-enhanced agents. This paper aims to streamline the concept of memory in RL by providing practical precise definitions of agent memory types, such as long-term versus short-term memory and declarative versus procedural memory, inspired by cognitive science. Using these definitions, we categorize different classes of agent memory, propose a robust experimental methodology for evaluating the memory capabilities of RL agents, and standardize evaluations. Furthermore, we empirically demonstrate the importance of adhering to the proposed methodology when evaluating different types of agent memory by conducting experiments with different RL agents and what its violation leads to.

揭開強化學習智能體記憶複雜性的方法：分類與評估

Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluation

摘要

Summary

Support