提議者-代理者-評估者（PAE）：基於模型互聯網代理的自主技能發現

摘要

對於廣泛具備能力和目標導向的智能體，例如數位世界中的網路瀏覽智能體和實體世界中的家庭機器人，其願景已經快速發展，這要歸功於基礎模型的泛化能力。這樣一個通才智能體需要具備龐大且多樣化的技能庫，例如在兩個旅行地點之間尋找方向，以及從網路上購買特定物品。如果每個技能都需要透過固定的一組人工註釋指示來手動指定，由於人工註釋指示的數量和多樣性，智能體的技能庫將會受到限制。在這項工作中，我們通過提出提案者-智能體-評估者（Proposer-Agent-Evaluator，PAE）來應對這一挑戰，這是一個有效的學習系統，使基礎模型智能體能夠在野外自主發現和練習技能。PAE 的核心是一個具有上下文感知能力的任務提案者，根據環境的上下文信息（例如用戶演示或僅僅是網站名稱）自主提出智能體練習的任務。然後，智能體政策通過思考和實際在現實世界中進行基於結果軌跡的操作來嘗試這些任務，這些軌跡由自主的基於VLM 的成功評估者評估。成功評估作為智能體通過強化學習來優化其政策的獎勵信號。我們在具有挑戰性的基於視覺的網頁導航上驗證了 PAE，使用了來自 WebVoyager 和 WebArena 的真實世界和自託管網站。據我們所知，這項工作代表了首個應用自主任務提案與強化學習的有效學習系統，能夠將真實世界的人工註釋基準泛化為具有 SOTA 性能的智能體。我們的開源檢查點和代碼可在 https://yanqval.github.io/PAE/ 找到。

English

The vision of a broadly capable and goal-directed agent, such as an Internet-browsing agent in the digital world and a household humanoid in the physical world, has rapidly advanced, thanks to the generalization capability of foundation models. Such a generalist agent needs to have a large and diverse skill repertoire, such as finding directions between two travel locations and buying specific items from the Internet. If each skill needs to be specified manually through a fixed set of human-annotated instructions, the agent's skill repertoire will necessarily be limited due to the quantity and diversity of human-annotated instructions. In this work, we address this challenge by proposing Proposer-Agent-Evaluator, an effective learning system that enables foundation model agents to autonomously discover and practice skills in the wild. At the heart of PAE is a context-aware task proposer that autonomously proposes tasks for the agent to practice with context information of the environment such as user demos or even just the name of the website itself for Internet-browsing agents. Then, the agent policy attempts those tasks with thoughts and actual grounded operations in the real world with resulting trajectories evaluated by an autonomous VLM-based success evaluator. The success evaluation serves as the reward signal for the agent to refine its policies through RL. We validate PAE on challenging vision-based web navigation, using both real-world and self-hosted websites from WebVoyager and WebArena.To the best of our knowledge, this work represents the first effective learning system to apply autonomous task proposal with RL for agents that generalizes real-world human-annotated benchmarks with SOTA performances. Our open-source checkpoints and code can be found in https://yanqval.github.io/PAE/

提議者-代理者-評估者（PAE）：基於模型互聯網代理的自主技能發現

Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agents

摘要

Summary

Support