臨床實體識別基準。

Named Clinical Entity Recognition Benchmark

October 7, 2024

作者: Wadood M Abdul, Marco AF Pimentel, Muhammad Umar Salman, Tathagata Raha, Clément Christophe, Praveen K Kanithi, Nasir Hayat, Ronnie Rajan, Shadab Khan

cs.AI

摘要

本技術報告介紹了一個名為臨床實體識別基準的基準，用於評估醫療保健中語言模型的性能，解決了從臨床敘事中提取結構化信息的關鍵自然語言處理（NLP）任務，以支持自動編碼、臨床試驗群體識別和臨床決策支持等應用。 排行榜提供了一個標準化平台，用於評估不同語言模型（包括編碼器和解碼器架構）在識別和分類多個醫學領域的臨床實體方面的能力。利用一個經過精心挑選的開放臨床數據集合，其中包含疾病、症狀、藥物、程序和實驗室測量等實體。重要的是，這些實體根據觀察性醫學結果合作夥伴關係（OMOP）通用數據模型進行了標準化，確保在不同醫療系統和數據集之間的一致性和互操作性，以及對模型性能的全面評估。模型的性能主要通過F1分數進行評估，並通過各種評估模式來提供對模型性能的全面洞察。報告還包括對迄今為止評估的模型的簡要分析，突出觀察到的趨勢和限制。 通過建立這個基準框架，排行榜旨在促進透明度，促進比較分析，並推動臨床實體識別任務的創新，解決醫療NLP中對強大評估方法的需求。

English

This technical report introduces a Named Clinical Entity Recognition Benchmark for evaluating language models in healthcare, addressing the crucial natural language processing (NLP) task of extracting structured information from clinical narratives to support applications like automated coding, clinical trial cohort identification, and clinical decision support. The leaderboard provides a standardized platform for assessing diverse language models, including encoder and decoder architectures, on their ability to identify and classify clinical entities across multiple medical domains. A curated collection of openly available clinical datasets is utilized, encompassing entities such as diseases, symptoms, medications, procedures, and laboratory measurements. Importantly, these entities are standardized according to the Observational Medical Outcomes Partnership (OMOP) Common Data Model, ensuring consistency and interoperability across different healthcare systems and datasets, and a comprehensive evaluation of model performance. Performance of models is primarily assessed using the F1-score, and it is complemented by various assessment modes to provide comprehensive insights into model performance. The report also includes a brief analysis of models evaluated to date, highlighting observed trends and limitations. By establishing this benchmarking framework, the leaderboard aims to promote transparency, facilitate comparative analyses, and drive innovation in clinical entity recognition tasks, addressing the need for robust evaluation methods in healthcare NLP.