ORID：器官區域資訊驅動的放射學報告生成框架

ORID: Organ-Regional Information Driven Framework for Radiology Report Generation

November 20, 2024

作者: Tiancheng Gu, Kaicheng Yang, Xiang An, Ziyong Feng, Dongnan Liu, Weidong Cai

cs.AI

摘要

放射學報告生成（RRG）的目標是根據放射影像自動生成關於疾病的連貫文本分析，從而減輕放射科醫師的工作量。目前基於人工智慧的RRG方法主要集中在對編碼器-解碼器模型架構的修改上。為了推進這些方法，本文引入了一個基於器官-區域信息驅動（ORID）的框架，可以有效整合多模態信息並減少來自無關器官的噪音影響。具體來說，基於LLaVA-Med，我們首先構建了一個與RRG相關的指導數據集，以提高器官區域診斷描述能力，並獲得LLaVA-Med-RRG。之後，我們提出了一個基於器官的跨模態融合模塊，以有效結合器官區域診斷描述和放射影像的信息。為了進一步減少無關器官對放射學報告生成的影響，我們引入了一個器官重要性係數分析模塊，利用圖神經網絡（GNN）來檢查每個器官區域的跨模態信息之間的相互聯繫。通過廣泛的實驗和與各種評估指標的最新方法進行比較，證明了我們提出的方法的優越性能。

English

The objective of Radiology Report Generation (RRG) is to automatically generate coherent textual analyses of diseases based on radiological images, thereby alleviating the workload of radiologists. Current AI-based methods for RRG primarily focus on modifications to the encoder-decoder model architecture. To advance these approaches, this paper introduces an Organ-Regional Information Driven (ORID) framework which can effectively integrate multi-modal information and reduce the influence of noise from unrelated organs. Specifically, based on the LLaVA-Med, we first construct an RRG-related instruction dataset to improve organ-regional diagnosis description ability and get the LLaVA-Med-RRG. After that, we propose an organ-based cross-modal fusion module to effectively combine the information from the organ-regional diagnosis description and radiology image. To further reduce the influence of noise from unrelated organs on the radiology report generation, we introduce an organ importance coefficient analysis module, which leverages Graph Neural Network (GNN) to examine the interconnections of the cross-modal information of each organ region. Extensive experiments an1d comparisons with state-of-the-art methods across various evaluation metrics demonstrate the superior performance of our proposed method.