BEATS：大型語言模型的偏差評估與測試套件

BEATS: Bias Evaluation and Assessment Test Suite for Large Language Models

March 31, 2025

作者: Alok Abhishek, Lisa Erickson, Tushar Bandopadhyay

cs.AI

摘要

在本研究中，我們引入了BEATS，這是一個用於評估大型語言模型（LLMs）中偏見、倫理、公平性和事實性的新框架。基於BEATS框架，我們提出了一個針對LLMs的偏見基準測試，該測試涵蓋了29個不同的指標。這些指標廣泛涵蓋了多種特徵，包括人口統計、認知和社會偏見，以及倫理推理、群體公平性和與事實相關的錯誤信息風險的衡量標準。這些指標使得我們能夠定量評估LLM生成的回應可能在多大程度上延續社會偏見，從而強化或擴大系統性不平等。要在這一基準測試中獲得高分，LLM必須在其回應中展現出極高的公平性，這使其成為負責任AI評估的嚴格標準。基於我們實驗數據的實證結果顯示，行業領先模型生成的輸出中有37.65%包含某種形式的偏見，這凸顯了在關鍵決策系統中使用這些模型的重大風險。BEATS框架和基準測試提供了一種可擴展且統計嚴謹的方法來對LLMs進行基準測試，診斷驅動偏見的因素，並制定緩解策略。通過BEATS框架，我們的目標是幫助開發更具社會責任感和倫理對齊的AI模型。

English

In this research, we introduce BEATS, a novel framework for evaluating Bias, Ethics, Fairness, and Factuality in Large Language Models (LLMs). Building upon the BEATS framework, we present a bias benchmark for LLMs that measure performance across 29 distinct metrics. These metrics span a broad range of characteristics, including demographic, cognitive, and social biases, as well as measures of ethical reasoning, group fairness, and factuality related misinformation risk. These metrics enable a quantitative assessment of the extent to which LLM generated responses may perpetuate societal prejudices that reinforce or expand systemic inequities. To achieve a high score on this benchmark a LLM must show very equitable behavior in their responses, making it a rigorous standard for responsible AI evaluation. Empirical results based on data from our experiment show that, 37.65\% of outputs generated by industry leading models contained some form of bias, highlighting a substantial risk of using these models in critical decision making systems. BEATS framework and benchmark offer a scalable and statistically rigorous methodology to benchmark LLMs, diagnose factors driving biases, and develop mitigation strategies. With the BEATS framework, our goal is to help the development of more socially responsible and ethically aligned AI models.