EmoAgent：評估與保障人機互動的心理健康安全

EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental Health Safety

April 13, 2025

作者: Jiahao Qiu, Yinghui He, Xinzhe Juan, Yiming Wang, Yuhan Liu, Zixin Yao, Yue Wu, Xun Jiang, Ling Yang, Mengdi Wang

cs.AI

摘要

大型語言模型（LLM）驅動的AI角色興起引發了安全隱憂，尤其對患有心理疾病的脆弱使用者而言。為應對這些風險，我們提出了EmoAgent，這是一個多代理AI框架，旨在評估並減輕人機互動中的心理健康危害。EmoAgent包含兩個組件：EmoEval模擬虛擬使用者，包括那些表現出心理脆弱性的個體，以評估與AI角色互動前後的心理健康變化。它採用臨床驗證的心理學與精神科評估工具（如PHQ-9、PDI、PANSS）來評估由LLM引發的心理風險。EmoGuard則作為中介，監控使用者的心理狀態，預測潛在傷害，並提供修正反饋以降低風險。在基於角色的熱門聊天機器人中進行的實驗顯示，情感投入的對話可能導致脆弱使用者的心理狀況惡化，超過34.4%的模擬案例中出現了心理狀態的退化。EmoGuard顯著降低了這些退化率，凸顯了其在確保更安全的人機互動中的重要作用。我們的程式碼已公開於：https://github.com/1akaman/EmoAgent。

English

The rise of LLM-driven AI characters raises safety concerns, particularly for vulnerable human users with psychological disorders. To address these risks, we propose EmoAgent, a multi-agent AI framework designed to evaluate and mitigate mental health hazards in human-AI interactions. EmoAgent comprises two components: EmoEval simulates virtual users, including those portraying mentally vulnerable individuals, to assess mental health changes before and after interactions with AI characters. It uses clinically proven psychological and psychiatric assessment tools (PHQ-9, PDI, PANSS) to evaluate mental risks induced by LLM. EmoGuard serves as an intermediary, monitoring users' mental status, predicting potential harm, and providing corrective feedback to mitigate risks. Experiments conducted in popular character-based chatbots show that emotionally engaging dialogues can lead to psychological deterioration in vulnerable users, with mental state deterioration in more than 34.4% of the simulations. EmoGuard significantly reduces these deterioration rates, underscoring its role in ensuring safer AI-human interactions. Our code is available at: https://github.com/1akaman/EmoAgent