斑马-大羊驼：一种面向上下文的大型语言模型，用于普及罕见疾病知识

Zebra-Llama: A Context-Aware Large Language Model for Democratizing Rare Disease Knowledge

November 4, 2024

作者: Karthik Soman, Andrew Langdon, Catalina Villouta, Chinmay Agrawal, Lashaw Salta, Braian Peetoom, Gianmarco Bellucci, Orion J Buske

cs.AI

摘要

罕见疾病在医疗保健中面临独特挑战，常常出现延迟诊断和信息碎片化的情况。在这些疾病中可靠知识的稀缺性为大型语言模型（LLMs）提供了一个独特挑战，支持临床管理并提供准确的患者信息，强调对这些“斑马”病例进行专注培训的必要性。我们提出了Zebra-Llama，这是一个专门的上下文感知语言模型，具有高精度的检索增强生成（RAG）能力，重点关注埃勒斯-丹洛斯综合征（EDS）作为我们的案例研究。EDS每5000人中就有1人患病，通过其多样的症状、多个亚型和不断发展的诊断标准展示了罕见疾病的复杂性。通过实施一种新颖的上下文感知微调方法，该方法是在医学文献、患者经验和临床资源中提取问题进行训练的，再加上经过专家精心策划的回答，Zebra-Llama在处理与EDS相关的查询方面展示了前所未有的能力。在从EDS患者和临床医生那里收集的真实问题测试集上，医学专家评估了两种模型生成的回答，揭示了Zebra-Llama在全面性（77.5% vs. 70.1%）、准确性（83.0% vs. 78.8%）、清晰度（74.7% vs. 72.0%）和引文可靠性（70.6% vs. 52.3%）方面相对于基础模型（Llama 3.1-8B-Instruct）的显著改进。作为一个开放资源发布，Zebra-Llama不仅提供了更易获取和可靠的EDS信息，还为开发其他罕见疾病的专门人工智能解决方案奠定了框架。这项工作代表了向罕见疾病管理民主化专家级知识迈出的关键一步，潜在地改变了医疗保健提供者和患者如何应对罕见疾病复杂领域的方式。

English

Rare diseases present unique challenges in healthcare, often suffering from delayed diagnosis and fragmented information landscapes. The scarcity of reliable knowledge in these conditions poses a distinct challenge for Large Language Models (LLMs) in supporting clinical management and delivering precise patient information underscoring the need for focused training on these 'zebra' cases. We present Zebra-Llama, a specialized context-aware language model with high precision Retrieval Augmented Generation (RAG) capability, focusing on Ehlers-Danlos Syndrome (EDS) as our case study. EDS, affecting 1 in 5,000 individuals, exemplifies the complexities of rare diseases with its diverse symptoms, multiple subtypes, and evolving diagnostic criteria. By implementing a novel context-aware fine-tuning methodology trained on questions derived from medical literature, patient experiences, and clinical resources, along with expertly curated responses, Zebra-Llama demonstrates unprecedented capabilities in handling EDS-related queries. On a test set of real-world questions collected from EDS patients and clinicians, medical experts evaluated the responses generated by both models, revealing Zebra-Llama's substantial improvements over base model (Llama 3.1-8B-Instruct) in thoroughness (77.5% vs. 70.1%), accuracy (83.0% vs. 78.8%), clarity (74.7% vs. 72.0%) and citation reliability (70.6% vs. 52.3%). Released as an open-source resource, Zebra-Llama not only provides more accessible and reliable EDS information but also establishes a framework for developing specialized AI solutions for other rare conditions. This work represents a crucial step towards democratizing expert-level knowledge in rare disease management, potentially transforming how healthcare providers and patients navigate the complex landscape of rare diseases.