ShieldAgent：通过可验证安全策略推理保护智能体

ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning

March 26, 2025

作者: Zhaorun Chen, Mintong Kang, Bo Li

cs.AI

摘要

基于基础模型的自主智能体已在各类现实应用中广泛部署。然而，这些智能体极易受到恶意指令和攻击的影响，可能导致隐私泄露和财务损失等严重后果。更为关键的是，由于智能体复杂且动态的特性，现有的大型语言模型防护措施并不适用。为应对这些挑战，我们提出了ShieldAgent，这是首个通过逻辑推理来确保其他受保护智能体行为轨迹明确遵循安全策略的防护智能体。具体而言，ShieldAgent首先从策略文档中提取可验证的规则，并将其构建为一组基于行为的概率规则电路，以此建立安全策略模型。针对受保护智能体的行为轨迹，ShieldAgent检索相关规则电路，并利用其丰富的工具库和可执行代码生成防护计划，进行形式化验证。此外，鉴于当前缺乏针对智能体的防护基准，我们引入了ShieldAgent-Bench，这是一个包含3,000对与安全相关的智能体指令和行为轨迹的数据集，这些数据通过最先进的攻击手段在6个网络环境和7个风险类别中收集。实验表明，ShieldAgent在ShieldAgent-Bench及三个现有基准测试上均达到了最先进水平，平均超越先前方法11.3%，召回率高达90.1%。同时，ShieldAgent将API调用减少了64.7%，推理时间缩短了58.2%，展现了其在保护智能体方面的高精度与高效性。

English

Autonomous agents powered by foundation models have seen widespread adoption across various real-world applications. However, they remain highly vulnerable to malicious instructions and attacks, which can result in severe consequences such as privacy breaches and financial losses. More critically, existing guardrails for LLMs are not applicable due to the complex and dynamic nature of agents. To tackle these challenges, we propose ShieldAgent, the first guardrail agent designed to enforce explicit safety policy compliance for the action trajectory of other protected agents through logical reasoning. Specifically, ShieldAgent first constructs a safety policy model by extracting verifiable rules from policy documents and structuring them into a set of action-based probabilistic rule circuits. Given the action trajectory of the protected agent, ShieldAgent retrieves relevant rule circuits and generates a shielding plan, leveraging its comprehensive tool library and executable code for formal verification. In addition, given the lack of guardrail benchmarks for agents, we introduce ShieldAgent-Bench, a dataset with 3K safety-related pairs of agent instructions and action trajectories, collected via SOTA attacks across 6 web environments and 7 risk categories. Experiments show that ShieldAgent achieves SOTA on ShieldAgent-Bench and three existing benchmarks, outperforming prior methods by 11.3% on average with a high recall of 90.1%. Additionally, ShieldAgent reduces API queries by 64.7% and inference time by 58.2%, demonstrating its high precision and efficiency in safeguarding agents.