教導多模式LLM理解心電圖影像

Teach Multimodal LLMs to Comprehend Electrocardiographic Images

October 21, 2024

作者: Ruoqi Liu, Yuelin Bai, Xiang Yue, Ping Zhang

cs.AI

摘要

心電圖（ECG）是評估心臟狀況的重要非侵入性診斷工具。現有的自動解讀方法存在泛化能力有限的問題，專注於狹窄範圍的心臟狀況，通常依賴原始生理信號，但在資源有限的環境中可能無法輕易取得，只能使用印刷或數位心電圖影像。最近多模式大型語言模型（MLLMs）的進步為應對這些挑戰帶來了機遇。然而，將MLLMs應用於心電圖影像解讀仍然具有挑戰性，因為缺乏指導調整數據集和用於量化評估的完善心電圖影像基準。為應對這些挑戰，我們引入了ECGInstruct，一個包含超過一百萬樣本的全面心電圖影像指導調整數據集，涵蓋來自多源數據的廣泛心電圖相關任務。利用ECGInstruct，我們開發了PULSE，一個針對心電圖影像理解而設計的MLLM。此外，我們精心編纂了ECGBench，一個新的評估基準，涵蓋九個不同數據集上的四個關鍵心電圖影像解讀任務。我們的實驗表明，PULSE創下了新的最先進水平，平均準確率提高了15%至30%，優於一般MLLMs。這項工作突顯了PULSE在臨床實踐中提升心電圖解讀能力的潛力。

English

The electrocardiogram (ECG) is an essential non-invasive diagnostic tool for assessing cardiac conditions. Existing automatic interpretation methods suffer from limited generalizability, focusing on a narrow range of cardiac conditions, and typically depend on raw physiological signals, which may not be readily available in resource-limited settings where only printed or digital ECG images are accessible. Recent advancements in multimodal large language models (MLLMs) present promising opportunities for addressing these challenges. However, the application of MLLMs to ECG image interpretation remains challenging due to the lack of instruction tuning datasets and well-established ECG image benchmarks for quantitative evaluation. To address these challenges, we introduce ECGInstruct, a comprehensive ECG image instruction tuning dataset of over one million samples, covering a wide range of ECG-related tasks from diverse data sources. Using ECGInstruct, we develop PULSE, an MLLM tailored for ECG image comprehension. In addition, we curate ECGBench, a new evaluation benchmark covering four key ECG image interpretation tasks across nine different datasets. Our experiments show that PULSE sets a new state-of-the-art, outperforming general MLLMs with an average accuracy improvement of 15% to 30%. This work highlights the potential of PULSE to enhance ECG interpretation in clinical practice.