臨床ModernBERT：一種高效且長上下文編碼器，專為生物醫學文本設計

Clinical ModernBERT: An efficient and long context encoder for biomedical text

April 4, 2025

作者: Simon A. Lee, Anthony Wu, Jeffrey N. Chiang

cs.AI

摘要

我們介紹了Clinical ModernBERT，這是一個基於Transformer的編碼器，預訓練於大規模生物醫學文獻、臨床筆記和醫學本體，整合了PubMed摘要、MIMIC IV臨床數據以及帶有文本描述的醫學代碼。基於當今最先進的自然語言文本編碼器ModernBERT，其架構升級包括旋轉位置嵌入（RoPE）、閃爍注意力（Flash Attention）以及擴展至8,192個標記的上下文長度，我們的模型特別針對生物醫學和臨床領域調整了這些創新。Clinical ModernBERT在生成語義豐富的表示方面表現卓越，尤其適合長上下文任務。我們通過分析其預訓練權重以及在全面的臨床自然語言處理基準上的實證評估來驗證這一點。

English

We introduce Clinical ModernBERT, a transformer based encoder pretrained on large scale biomedical literature, clinical notes, and medical ontologies, incorporating PubMed abstracts, MIMIC IV clinical data, and medical codes with their textual descriptions. Building on ModernBERT the current state of the art natural language text encoder featuring architectural upgrades such as rotary positional embeddings (RoPE), Flash Attention, and extended context length up to 8,192 tokens our model adapts these innovations specifically for biomedical and clinical domains. Clinical ModernBERT excels at producing semantically rich representations tailored for long context tasks. We validate this both by analyzing its pretrained weights and through empirical evaluation on a comprehensive suite of clinical NLP benchmarks.