教授多模态LLMs理解心电图像

Teach Multimodal LLMs to Comprehend Electrocardiographic Images

October 21, 2024

作者: Ruoqi Liu, Yuelin Bai, Xiang Yue, Ping Zhang

cs.AI

摘要

心电图（ECG）是评估心脏状况的重要非侵入性诊断工具。现有的自动解释方法存在泛化能力有限的问题，专注于狭窄范围的心脏状况，并且通常依赖于原始生理信号，这些信号在资源有限的环境中可能无法直接获取，只能获得打印或数字化的心电图像。最近多模态大型语言模型（MLLMs）的进展为解决这些挑战提供了希望。然而，将MLLMs应用于心电图像解释仍然具有挑战性，因为缺乏指导调整数据集和用于定量评估的完善的心电图像基准。为了解决这些挑战，我们引入了ECGInstruct，这是一个包含一百多万样本的全面的心电图像指导调整数据集，涵盖了来自不同数据源的广泛心电图相关任务。利用ECGInstruct，我们开发了PULSE，这是一个专为心电图像理解量身定制的MLLM。此外，我们策划了ECGBench，一个新的评估基准，涵盖了九个不同数据集中的四个关键心电图像解释任务。我们的实验表明，PULSE取得了新的最先进水平，平均准确率提高了15%至30%，胜过了通用MLLMs。这项工作突显了PULSE在临床实践中提升心电图解释的潜力。

English

The electrocardiogram (ECG) is an essential non-invasive diagnostic tool for assessing cardiac conditions. Existing automatic interpretation methods suffer from limited generalizability, focusing on a narrow range of cardiac conditions, and typically depend on raw physiological signals, which may not be readily available in resource-limited settings where only printed or digital ECG images are accessible. Recent advancements in multimodal large language models (MLLMs) present promising opportunities for addressing these challenges. However, the application of MLLMs to ECG image interpretation remains challenging due to the lack of instruction tuning datasets and well-established ECG image benchmarks for quantitative evaluation. To address these challenges, we introduce ECGInstruct, a comprehensive ECG image instruction tuning dataset of over one million samples, covering a wide range of ECG-related tasks from diverse data sources. Using ECGInstruct, we develop PULSE, an MLLM tailored for ECG image comprehension. In addition, we curate ECGBench, a new evaluation benchmark covering four key ECG image interpretation tasks across nine different datasets. Our experiments show that PULSE sets a new state-of-the-art, outperforming general MLLMs with an average accuracy improvement of 15% to 30%. This work highlights the potential of PULSE to enhance ECG interpretation in clinical practice.