从RAG到记忆：面向大语言模型的非参数化持续学习

From RAG to Memory: Non-Parametric Continual Learning for Large Language Models

February 20, 2025

作者: Bernal Jiménez Gutiérrez, Yiheng Shu, Weijian Qi, Sizhe Zhou, Yu Su

cs.AI

摘要

我们持续获取、组织并利用知识的能力，是人类智能的关键特征，也是人工智能系统必须模拟以实现其全部潜力的核心要素。鉴于大型语言模型（LLMs）在持续学习方面面临的挑战，检索增强生成（RAG）已成为引入新信息的主导方式。然而，RAG对向量检索的依赖限制了其模拟人类长期记忆动态与互联特性的能力。近期RAG方法通过结合知识图谱等多种结构来增强向量嵌入，旨在弥补理解力与关联性方面的不足。但它们在基础事实记忆任务上的表现却显著低于标准RAG。针对这一非预期的性能下降，我们提出了HippoRAG 2框架，该框架在事实记忆、理解力及关联记忆任务上全面超越标准RAG。HippoRAG 2基于HippoRAG中使用的个性化PageRank算法，通过更深层次的段落整合及更高效的LLM在线应用加以强化。这一组合使RAG系统更接近人类长期记忆的效能，在关联记忆任务上较当前最先进的嵌入模型提升了7%，同时展现出更优的事实知识与理解记忆能力。本工作为LLMs的非参数持续学习开辟了道路。我们的代码与数据将在https://github.com/OSU-NLP-Group/HippoRAG 发布。

English

Our ability to continuously acquire, organize, and leverage knowledge is a key feature of human intelligence that AI systems must approximate to unlock their full potential. Given the challenges in continual learning with large language models (LLMs), retrieval-augmented generation (RAG) has become the dominant way to introduce new information. However, its reliance on vector retrieval hinders its ability to mimic the dynamic and interconnected nature of human long-term memory. Recent RAG approaches augment vector embeddings with various structures like knowledge graphs to address some of these gaps, namely sense-making and associativity. However, their performance on more basic factual memory tasks drops considerably below standard RAG. We address this unintended deterioration and propose HippoRAG 2, a framework that outperforms standard RAG comprehensively on factual, sense-making, and associative memory tasks. HippoRAG 2 builds upon the Personalized PageRank algorithm used in HippoRAG and enhances it with deeper passage integration and more effective online use of an LLM. This combination pushes this RAG system closer to the effectiveness of human long-term memory, achieving a 7% improvement in associative memory tasks over the state-of-the-art embedding model while also exhibiting superior factual knowledge and sense-making memory capabilities. This work paves the way for non-parametric continual learning for LLMs. Our code and data will be released at https://github.com/OSU-NLP-Group/HippoRAG.