参数空间中的技能扩展和组合

Skill Expansion and Composition in Parameter Space

February 9, 2025

作者: Tenglong Liu, Jianxiong Li, Yinan Zheng, Haoyi Niu, Yixing Lan, Xin Xu, Xianyuan Zhan

cs.AI

摘要

人类擅长重复利用先前知识来应对新挑战，并在解决问题的过程中发展技能。这种范式在自主代理的发展中变得越来越流行，因为它开发了能够像人类一样对新挑战进行自我演化的系统。然而，先前的方法在扩展新技能时存在训练效率有限的问题，并未充分利用先前知识来促进新任务的学习。在本文中，我们提出了参数化技能扩展与组合（PSEC），这是一个新框架，旨在通过保持可管理的技能库，通过迭代演化代理的能力，高效地应对新挑战。这个库可以逐步将技能基元作为即插即用的低秩适应（LoRA）模块集成到参数高效微调中，促进高效灵活的技能扩展。这种结构还使得能够在参数空间中直接组合技能，通过合并编码不同技能的LoRA模块，利用跨技能的共享信息来有效地设计新技能。基于此，我们提出了一个上下文感知模块，动态激活不同技能以协同处理新任务。在D4RL、DSRL基准和DeepMind控制套件上的结果显示，PSEC在利用先前知识高效应对新挑战以及扩展其技能库以演化能力方面表现出优越能力。项目网站：https://ltlhuuu.github.io/PSEC/。

English

Humans excel at reusing prior knowledge to address new challenges and developing skills while solving problems. This paradigm becomes increasingly popular in the development of autonomous agents, as it develops systems that can self-evolve in response to new challenges like human beings. However, previous methods suffer from limited training efficiency when expanding new skills and fail to fully leverage prior knowledge to facilitate new task learning. In this paper, we propose Parametric Skill Expansion and Composition (PSEC), a new framework designed to iteratively evolve the agents' capabilities and efficiently address new challenges by maintaining a manageable skill library. This library can progressively integrate skill primitives as plug-and-play Low-Rank Adaptation (LoRA) modules in parameter-efficient finetuning, facilitating efficient and flexible skill expansion. This structure also enables the direct skill compositions in parameter space by merging LoRA modules that encode different skills, leveraging shared information across skills to effectively program new skills. Based on this, we propose a context-aware module to dynamically activate different skills to collaboratively handle new tasks. Empowering diverse applications including multi-objective composition, dynamics shift, and continual policy shift, the results on D4RL, DSRL benchmarks, and the DeepMind Control Suite show that PSEC exhibits superior capacity to leverage prior knowledge to efficiently tackle new challenges, as well as expand its skill libraries to evolve the capabilities. Project website: https://ltlhuuu.github.io/PSEC/.