ProSA：評估與理解LLM對提示的敏感性

ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs

October 16, 2024

作者: Jingming Zhuo, Songyang Zhang, Xinyu Fang, Haodong Duan, Dahua Lin, Kai Chen

cs.AI

摘要

大型語言模型（LLMs）展示了在各種任務上令人印象深刻的能力，但它們的表現對使用的提示非常敏感。這種變異性對於準確評估和用戶滿意度構成挑戰。目前的研究經常忽略了實例級提示變化及其對主觀評估的影響。為解決這些缺陷，我們介紹了ProSA，這是一個旨在評估和理解LLMs中提示敏感性的框架。ProSA結合了一個新穎的敏感度指標PromptSensiScore，並利用解碼置信度來闡明潛在機制。我們的廣泛研究跨越多個任務，揭示了提示敏感性在數據集和模型之間波動，較大的模型表現出增強的穩健性。我們觀察到少樣本示例可以緩解這種敏感性問題，主觀評估也容易受到提示敏感性的影響，特別是在複雜的、注重推理的任務中。此外，我們的發現表明，模型置信度較高與提示穩健性增加呈正相關。我們相信這項工作將成為研究LLMs提示敏感性的有用工具。該項目已在以下網址釋出：https://github.com/open-compass/ProSA。

English

Large language models (LLMs) have demonstrated impressive capabilities across various tasks, but their performance is highly sensitive to the prompts utilized. This variability poses challenges for accurate assessment and user satisfaction. Current research frequently overlooks instance-level prompt variations and their implications on subjective evaluations. To address these shortcomings, we introduce ProSA, a framework designed to evaluate and comprehend prompt sensitivity in LLMs. ProSA incorporates a novel sensitivity metric, PromptSensiScore, and leverages decoding confidence to elucidate underlying mechanisms. Our extensive study, spanning multiple tasks, uncovers that prompt sensitivity fluctuates across datasets and models, with larger models exhibiting enhanced robustness. We observe that few-shot examples can alleviate this sensitivity issue, and subjective evaluations are also susceptible to prompt sensitivities, particularly in complex, reasoning-oriented tasks. Furthermore, our findings indicate that higher model confidence correlates with increased prompt robustness. We believe this work will serve as a helpful tool in studying prompt sensitivity of LLMs. The project is released at: https://github.com/open-compass/ProSA .