ClinicalBench：LLM能否在临床预测中击败传统的机器学习模型？

ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?

November 10, 2024

作者: Canyu Chen, Jian Yu, Shan Chen, Che Liu, Zhongwei Wan, Danielle Bitterman, Fei Wang, Kai Shu

cs.AI

摘要

大型语言模型（LLMs）在医学文本处理任务和医学执照考试方面具有优越的能力，因此对于革新当前临床系统具有巨大潜力。与此同时，传统的机器学习模型，如支持向量机（SVM）和XGBoost，在临床预测任务中仍然被主要采用。一个新兴的问题是，LLMs能否在临床预测中击败传统的机器学习模型？因此，我们建立了一个新的基准测试工具ClinicalBench，全面研究通用型和医学LLMs的临床预测建模能力，并将它们与传统机器学习模型进行比较。ClinicalBench涵盖了三个常见的临床预测任务、两个数据库、14个通用型LLMs、8个医学LLMs和11个传统机器学习模型。通过广泛的实证研究，我们发现，无论是通用型还是医学LLMs，即使在不同的模型规模、不同的提示或微调策略下，仍然无法在临床预测中击败传统的机器学习模型，这揭示了它们在临床推理和决策方面潜在的不足。我们呼吁从业者在临床应用中谨慎使用LLMs。ClinicalBench可以用于弥合LLMs在医疗保健领域发展和实际临床实践之间的差距。

English

Large Language Models (LLMs) hold great promise to revolutionize current clinical systems for their superior capacities on medical text processing tasks and medical licensing exams. Meanwhile, traditional ML models such as SVM and XGBoost have still been mainly adopted in clinical prediction tasks. An emerging question is Can LLMs beat traditional ML models in clinical prediction? Thus, we build a new benchmark ClinicalBench to comprehensively study the clinical predictive modeling capacities of both general-purpose and medical LLMs, and compare them with traditional ML models. ClinicalBench embraces three common clinical prediction tasks, two databases, 14 general-purpose LLMs, 8 medical LLMs, and 11 traditional ML models. Through extensive empirical investigation, we discover that both general-purpose and medical LLMs, even with different model scales, diverse prompting or fine-tuning strategies, still cannot beat traditional ML models in clinical prediction yet, shedding light on their potential deficiency in clinical reasoning and decision-making. We call for caution when practitioners adopt LLMs in clinical applications. ClinicalBench can be utilized to bridge the gap between LLMs' development for healthcare and real-world clinical practice.