Utility of Multimodal Large Language Models in Analyzing Chest X-Rays with Incomplete Contextual Information

Abstract

Objectives: Large language models (LLMs) are increasingly used in clinical practice, but their performance can deteriorate whenradiology reports are incomplete. We evaluated whether multimodal LLMs (integrating text and images) could enhance accuracyand interpretability in chest radiography reports, thereby improving their utility for clinical decision support. Specifically,we aimed to assess the robustness of LLMs in generating accurate impressions from chest radiography reports when providedwith incomplete data, and whether multimodal input could mitigate performance loss. Methods: We analyzed 300 radiologyimage–report pairs from the MIMIC-CXR database. Three LLMs—OpenFlamingo, MedFlamingo, IDEFICS—were tested intext-only and multimodal formats. Chest X-ray impressions were generated from complete text reports and then regeneratedafter systematically removing 20%, 50%, and 80% of the text. The effect of adding images was evaluated using chest X-rays, andmodel performance was compared using three statistical methods. Hallucination rates were quantified. Results: In the textonlysetting, OpenFlamingo, MedFlamingo, and IDEFICS demonstrated comparable performance (ROUGE-L: 0.23 vs. 0.21 vs. 0.21; F1RadGraph: 0.20 vs. 0.16 vs. 0.16; F1CheXbert: 0.49 vs. 0.41 vs. 0.41), with OpenFlamingo performing best on completetext (p < 0.001). All models exhibited performance decline with incomplete data. However, multimodal input significantly improvedthe performance of MedFlamingo and IDEFICS (p < 0.001), equaling or surpassing OpenFlamingo even under incompletetext conditions. Regarding hallucination, MedFlamingo showed a lower false-negative rate in multimodal compared withunimodal use, while false-positive rates were similar. Conclusions: LLMs may produce suboptimal outputs when radiologydata are incomplete, but multimodal LLMs enhance reliability and may strengthen clinical decision-making support.