WorldMedQA-V：一個多語言、多模態醫學檢查數據集，用於多模態語言模型評估。

WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation

October 16, 2024

作者: João Matos, Shan Chen, Siena Placino, Yingya Li, Juan Carlos Climent Pardo, Daphna Idan, Takeshi Tohyama, David Restrepo, Luis F. Nakayama, Jose M. M. Pascual-Leone, Guergana Savova, Hugo Aerts, Leo A. Celi, A. Ian Wong, Danielle S. Bitterman, Jack Gallifant

cs.AI

摘要

多模態/視覺語言模型（VLMs）越來越多地被部署在全球的醫療環境中，這需要強大的基準來確保其安全性、效能和公平性。源自國家醫學考試的多選問題和答案（QA）數據集長期以來一直是有價值的評估工具，但現有數據集主要僅限於文本，並且僅提供有限的語言和國家。為應對這些挑戰，我們提出了WorldMedQA-V，這是一個更新的多語言、多模態基準數據集，旨在評估醫療領域中的VLMs。WorldMedQA-V 包括來自巴西、以色列、日本和西班牙四個國家的 568 個標記的多選QA，配對了 568 張醫學圖像，分別涵蓋原始語言和由本地臨床醫生驗證的英文翻譯。提供了常見開源和封閉源模型的基準性能，以當地語言和英文翻譯呈現，並提供模型的圖像有無。WorldMedQA-V基準旨在更好地將AI系統與其部署的多樣化醫療環境相匹配，促進更具公平性、有效性和代表性的應用。

English

Multimodal/vision language models (VLMs) are increasingly being deployed in healthcare settings worldwide, necessitating robust benchmarks to ensure their safety, efficacy, and fairness. Multiple-choice question and answer (QA) datasets derived from national medical examinations have long served as valuable evaluation tools, but existing datasets are largely text-only and available in a limited subset of languages and countries. To address these challenges, we present WorldMedQA-V, an updated multilingual, multimodal benchmarking dataset designed to evaluate VLMs in healthcare. WorldMedQA-V includes 568 labeled multiple-choice QAs paired with 568 medical images from four countries (Brazil, Israel, Japan, and Spain), covering original languages and validated English translations by native clinicians, respectively. Baseline performance for common open- and closed-source models are provided in the local language and English translations, and with and without images provided to the model. The WorldMedQA-V benchmark aims to better match AI systems to the diverse healthcare environments in which they are deployed, fostering more equitable, effective, and representative applications.