AndroidLab：訓練和系統性基準測試Android自主代理程式。

AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents

October 31, 2024

作者: Yifan Xu, Xiao Liu, Xueqiao Sun, Siyi Cheng, Hao Yu, Hanyu Lai, Shudan Zhang, Dan Zhang, Jie Tang, Yuxiao Dong

cs.AI

摘要

自主代理在與現實世界互動中變得越來越重要。特別是，Android代理最近成為一種經常提及的互動方法。然而，現有用於訓練和評估Android代理的研究缺乏對開源和封閉源模型的系統性研究。在這項工作中，我們提出了AndroidLab作為一個系統化的Android代理框架。它包括一個具有不同模態、動作空間和可重現基準的操作環境。它支持在相同動作空間中的大型語言模型（LLMs）和多模型模型（LMMs）。AndroidLab基準包括預定義的Android虛擬設備和跨九個應用程序的138個任務。通過使用AndroidLab環境，我們開發了一個Android指令數據集，並訓練了六個開源的LLMs和LMMs，將LLMs的平均成功率從4.59%提高到21.50%，將LMMs的平均成功率從1.93%提高到13.28%。AndroidLab是開源的，並且可以在https://github.com/THUDM/Android-Lab 上公開獲取。

English

Autonomous agents have become increasingly important for interacting with the real world. Android agents, in particular, have been recently a frequently-mentioned interaction method. However, existing studies for training and evaluating Android agents lack systematic research on both open-source and closed-source models. In this work, we propose AndroidLab as a systematic Android agent framework. It includes an operation environment with different modalities, action space, and a reproducible benchmark. It supports both large language models (LLMs) and multimodal models (LMMs) in the same action space. AndroidLab benchmark includes predefined Android virtual devices and 138 tasks across nine apps built on these devices. By using the AndroidLab environment, we develop an Android Instruction dataset and train six open-source LLMs and LMMs, lifting the average success rates from 4.59% to 21.50% for LLMs and from 1.93% to 13.28% for LMMs. AndroidLab is open-sourced and publicly available at https://github.com/THUDM/Android-Lab.