Agent-SafetyBench：评估LLM Agent的安全性

Agent-SafetyBench: Evaluating the Safety of LLM Agents

December 19, 2024

作者: Zhexin Zhang, Shiyao Cui, Yida Lu, Jingzhuo Zhou, Junxiao Yang, Hongning Wang, Minlie Huang

cs.AI

摘要

随着大型语言模型（LLMs）越来越多地被部署为代理人，它们整合到互动环境和工具使用中引入了新的安全挑战，超越了与模型本身相关的挑战。然而，缺乏用于评估代理人安全性的全面基准构成了有效评估和进一步改进的重要障碍。在本文中，我们介绍Agent-SafetyBench，这是一个旨在评估LLM代理人安全性的全面基准。Agent-SafetyBench包括349个互动环境和2,000个测试用例，评估8类安全风险，并涵盖10种常见的失败模式，这些模式经常在不安全的互动中遇到。我们对16个流行的LLM代理人进行评估，结果令人担忧：没有一个代理人的安全得分超过60％。这突显了LLM代理人中存在的重大安全挑战，并强调了改进的重大需求。通过定量分析，我们确定了关键的失败模式，并总结了当前LLM代理人中两个基本的安全缺陷：缺乏鲁棒性和缺乏风险意识。此外，我们的研究结果表明，仅依赖于防御提示是不足以解决这些安全问题的，强调了更先进和更健壮策略的必要性。我们在https://github.com/thu-coai/Agent-SafetyBench发布了Agent-SafetyBench，以促进代理人安全评估和改进领域的进一步研究和创新。

English

As large language models (LLMs) are increasingly deployed as agents, their integration into interactive environments and tool use introduce new safety challenges beyond those associated with the models themselves. However, the absence of comprehensive benchmarks for evaluating agent safety presents a significant barrier to effective assessment and further improvement. In this paper, we introduce Agent-SafetyBench, a comprehensive benchmark designed to evaluate the safety of LLM agents. Agent-SafetyBench encompasses 349 interaction environments and 2,000 test cases, evaluating 8 categories of safety risks and covering 10 common failure modes frequently encountered in unsafe interactions. Our evaluation of 16 popular LLM agents reveals a concerning result: none of the agents achieves a safety score above 60%. This highlights significant safety challenges in LLM agents and underscores the considerable need for improvement. Through quantitative analysis, we identify critical failure modes and summarize two fundamental safety detects in current LLM agents: lack of robustness and lack of risk awareness. Furthermore, our findings suggest that reliance on defense prompts alone is insufficient to address these safety issues, emphasizing the need for more advanced and robust strategies. We release Agent-SafetyBench at https://github.com/thu-coai/Agent-SafetyBench to facilitate further research and innovation in agent safety evaluation and improvement.