In-Context Learning with Large Language Models: A Simple and Effective Approach to Improve Radiology Report Labeling

Abstract

Objectives: This study assessed the effectiveness of in-context learning using Generative Pre-trained Transformer-4 (GPT-4)for labeling radiology reports. Methods: In this retrospective study, radiology reports were obtained from the Medical InformationMart for Intensive Care III database. Two structured prompts—the “basic prompt” and the “in-context prompt”—were compared. An optimization experiment was conducted to assess consistency and the occurrence of output format errors. The primary labeling experiments were performed on 200 unseen head computed tomography (CT) reports for multilabelclassification of predefined labels (Experiment 1) and on 400 unseen abdominal CT reports for multi-label classificationof actionable findings (Experiment 2). Results: The inter-reader accuracies in Experiments 1 and 2 were 0.93 and 0.84,respectively. For multi-label classification of head CT reports (Experiment 1), the in-context prompt led to notable increasesin F1-scores for the “foreign body” and “mass” labels (gains of 0.66 and 0.22, respectively). However, improvements for otherlabels were modest. In multi-label classification of abdominal CT reports (Experiment 2), in-context prompts producedsubstantial improvements in F1-scores across all labels compared to basic prompts. Providing context equipped the modelwith domain-specific knowledge and helped align its existing knowledge, thereby improving performance. Conclusions: Incontextlearning with GPT-4 consistently improved performance in labeling radiology reports. This approach is particularlyeffective for subjective labeling tasks and allows the model to align its criteria with those of human annotators for objectivelabeling. This practical