ORID：面向器官区域信息的放射学报告生成框架

ORID: Organ-Regional Information Driven Framework for Radiology Report Generation

November 20, 2024

作者: Tiancheng Gu, Kaicheng Yang, Xiang An, Ziyong Feng, Dongnan Liu, Weidong Cai

cs.AI

摘要

放射学报告生成（RRG）的目标是基于放射影像自动生成关于疾病的连贯文本分析，从而减轻放射科医师的工作量。目前基于人工智能的RRG方法主要集中在修改编码器-解码器模型架构上。为了推进这些方法，本文引入了一种基于器官-区域信息驱动（ORID）的框架，可以有效整合多模态信息并减少来自无关器官的噪音影响。具体而言，基于LLaVA-Med，我们首先构建了一个与RRG相关的指导数据集，以提高器官-区域诊断描述能力，并得到LLaVA-Med-RRG。之后，我们提出了一个基于器官的跨模态融合模块，有效地结合了器官-区域诊断描述和放射影像的信息。为了进一步减少无关器官对放射学报告生成的影响，我们引入了一个器官重要性系数分析模块，利用图神经网络（GNN）来检查每个器官区域的跨模态信息之间的相互关系。通过广泛的实验和与各种评估指标的比较，我们提出的方法表现出优越的性能。

English

The objective of Radiology Report Generation (RRG) is to automatically generate coherent textual analyses of diseases based on radiological images, thereby alleviating the workload of radiologists. Current AI-based methods for RRG primarily focus on modifications to the encoder-decoder model architecture. To advance these approaches, this paper introduces an Organ-Regional Information Driven (ORID) framework which can effectively integrate multi-modal information and reduce the influence of noise from unrelated organs. Specifically, based on the LLaVA-Med, we first construct an RRG-related instruction dataset to improve organ-regional diagnosis description ability and get the LLaVA-Med-RRG. After that, we propose an organ-based cross-modal fusion module to effectively combine the information from the organ-regional diagnosis description and radiology image. To further reduce the influence of noise from unrelated organs on the radiology report generation, we introduce an organ importance coefficient analysis module, which leverages Graph Neural Network (GNN) to examine the interconnections of the cross-modal information of each organ region. Extensive experiments an1d comparisons with state-of-the-art methods across various evaluation metrics demonstrate the superior performance of our proposed method.