EmoAgent：评估与保障人机交互中的心理健康安全

EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental Health Safety

April 13, 2025

作者: Jiahao Qiu, Yinghui He, Xinzhe Juan, Yiming Wang, Yuhan Liu, Zixin Yao, Yue Wu, Xun Jiang, Ling Yang, Mengdi Wang

cs.AI

摘要

大型语言模型（LLM）驱动的AI角色兴起引发了安全担忧，尤其针对心理障碍等脆弱人群。为应对这些风险，我们提出了EmoAgent，一个多智能体AI框架，旨在评估并缓解人机交互中的心理健康隐患。EmoAgent包含两大组件：EmoEval通过模拟虚拟用户，包括展现心理脆弱特征的个体，来评估与AI角色互动前后的心理健康变化。它采用临床验证的心理与精神病学评估工具（如PHQ-9、PDI、PANSS）来量化LLM引发的心理风险。EmoGuard则作为中介，实时监控用户心理状态，预测潜在伤害，并提供纠正性反馈以降低风险。在主流角色型聊天机器人中的实验表明，情感投入的对话可能导致脆弱用户心理状况恶化，超过34.4%的模拟案例出现心理状态下滑。EmoGuard显著降低了这一恶化比例，凸显了其在确保人机交互安全中的关键作用。我们的代码已开源，访问地址：https://github.com/1akaman/EmoAgent。

English

The rise of LLM-driven AI characters raises safety concerns, particularly for vulnerable human users with psychological disorders. To address these risks, we propose EmoAgent, a multi-agent AI framework designed to evaluate and mitigate mental health hazards in human-AI interactions. EmoAgent comprises two components: EmoEval simulates virtual users, including those portraying mentally vulnerable individuals, to assess mental health changes before and after interactions with AI characters. It uses clinically proven psychological and psychiatric assessment tools (PHQ-9, PDI, PANSS) to evaluate mental risks induced by LLM. EmoGuard serves as an intermediary, monitoring users' mental status, predicting potential harm, and providing corrective feedback to mitigate risks. Experiments conducted in popular character-based chatbots show that emotionally engaging dialogues can lead to psychological deterioration in vulnerable users, with mental state deterioration in more than 34.4% of the simulations. EmoGuard significantly reduces these deterioration rates, underscoring its role in ensuring safer AI-human interactions. Our code is available at: https://github.com/1akaman/EmoAgent