Efficacy of large language models and their potential in Obstetrics and Gynecology education
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Objective The performance of large language models (LLMs) and their potential utility in obstetric and gynecological education are topics of ongoing debate. This study aimed to contribute to this discussion by examining the recent advancements in LLM technology and their transformative potential in artificial intelligence. Methods This study assessed the performance of generative pre-trained transformer (GPT)-3.5 and -4 in understanding clinical information, as well as its potential implications for obstetric and gynecological education. Obstetrics and gynecology residents at three hospitals underwent an annual promotional examination, from which 116 of the 170 questions over 4 years (2020-2023) were analyzed, excluding 54 questions with images. The scores achieved by GPT-3.5, -4, and the 100 residents were compared. Results The average scores across all 4 years for GPT-3.5 and -4 were 38.79 (standard deviation [SD], 5.65) and 79.31 (SD, 3.67), respectively. For groups first-year resident, second-year resident, and third-year resident, the cumulative annual average scores were 79.12 (SD, 9.00), 80.95 (SD, 5.86), and 83.60 (SD, 6.82), respectively. No statistically significant differences were observed between the scores of GPT-4.0 and those of the residents. When analyzing questions specific to obstetrics, the average scores for GPT-3.5 and -4.0 were 33.44 (SD, 10.18) and 90.22 (SD, 7.68), respectively. Conclusion GPT-4 demonstrated exceptional performance in obstetrics, different types of data interpretation, and problem solving, showcasing the potential utility of LLMs in these areas. However, acknowledging the constraints of LLMs is crucial and their utilization should augment human expertise and discernment.
Abstract
Objective The performance of large language models (LLMs) and their potential utility in obstetric and gynecological education are topics of ongoing debate. This study aimed to contribute to this discussion by examining the recent advancements in LLM technology and their transformative potential in artificial intelligence. Methods This study assessed the performance of generative pre-trained transformer (GPT)-3.5 and -4 in understanding clinical information, as well as its potential implications for obstetric and gynecological education. Obstetrics and gynecology residents at three hospitals underwent an annual promotional examination, from which 116 of the 170 questions over 4 years (2020-2023) were analyzed, excluding 54 questions with images. The scores achieved by GPT-3.5, -4, and the 100 residents were compared. Results The average scores across all 4 years for GPT-3.5 and -4 were 38.79 (standard deviation [SD], 5.65) and 79.31 (SD, 3.67), respectively. For groups first-year resident, second-year resident, and third-year resident, the cumulative annual average scores were 79.12 (SD, 9.00), 80.95 (SD, 5.86), and 83.60 (SD, 6.82), respectively. No statistically significant differences were observed between the scores of GPT-4.0 and those of the residents. When analyzing questions specific to obstetrics, the average scores for GPT-3.5 and -4.0 were 33.44 (SD, 10.18) and 90.22 (SD, 7.68), respectively. Conclusion GPT-4 demonstrated exceptional performance in obstetrics, different types of data interpretation, and problem solving, showcasing the potential utility of LLMs in these areas. However, acknowledging the constraints of LLMs is crucial and their utilization should augment human expertise and discernment.
참고문헌 (20)
- Stokel-Walker C, "What ChatGPT and generative AI mean for science", Nature, 614, pp.214~216 (2023)
- Kung TH, "Performance of ChatGPT on USMLE : potential for AI-assisted medical education using large language models", PLOS Digit Health, 2, pp.e0000198~ (2023)
- Mbakwe AB, "ChatGPT passing USMLE shines a spotlight on the flaws of medical education", PLOS Digit Health, 2, pp.e0000205~ (2023)
- Oh N, "ChatGPT goes to the operating room : evaluating GPT-4 performance and its potential in surgical education and training in the era of large language models", Ann Surg Treat Res, 104, pp.269~273 (2023)
- Yaneva V, "Examining ChatGPT performance on USMLE sample items and implications for assessment", Acad Med, 99, pp.192~197 (2024)
- Watari T, "Performance comparison of Chat-GPT-4 and Japanese medical residents in the general medicine in-training examination : comparison study", JMIR Med Educ, 9, pp.e52202~ (2023)
- Ali R, "Performance of ChatGPT and GPT-4 on neurosurgery written board examinations", Neurosurgery, 93, pp.1353~1365 (2023)
- Chen KT, "Current practices and perspectives on clerkship grading in obstetrics and gynecology", Am J Obstet Gynecol, 230, pp.97~ (2024)
- Wartman SA, "Medical education must move from the information age to the age of artificial intelligence", Acad Med, 93, pp.1107~1109 (2018)
- Ahn KH, "Artificial intelligence in obstetrics", Obstet Gynecol Sci, 65, pp.113~124 (2022)
- Ong H, "GPT technology to help address longstanding barriers to care in free medical clinics", Ann Biomed Eng, 51, pp.1906~1909 (2023)
- Bhattarai K, "Leveraging GPT-4 for identifying cancer phenotypes in electronic health records : a performance comparison between GPT-4, GPT-3. 5-turbo, Flan-T5 and spaCy’s rule-based & machine learning-based methods", JAMIA Open, 7, pp.ooae060~ (2024)
- Phung A, "Multispecialty trainee perspective : the journey toward competency-based graduate medical education in the United States", J Grad Med Educ, 15, pp.617~622 (2023)
- Kapadia MR, "Being affable, available, and able is not enough : prioritizing surgeon-patient communication", JAMA Surg, 155, pp.277~278 (2020)
- Abd-Alrazaq A, "Large language models in medical education : opportunities, challenges, and future directions", JMIR Med Educ, 9, pp.e48291~ (2023)
- Jamal A, "Integrating ChatGPT in medical education : adapting curricula to cultivate competent physicians for the AI Era", Cureus, 15, pp.e43036~ (2023)
- Han ER, "Medical education trends for future physicians in the era of advanced technology and artificial intelligence : an integrative review", BMC Med Educ, 19, pp.460~ (2019)
- Sharma A, "Artificial intelligence chatbots : addressing the stochastic parrots in medical science", Nucl Med Commun, 44, pp.831~833 (2023)
- Luo R, "BioGPT : generative pre-trained transformer for biomedical text generation and mining", Brief Bioinform, 23, pp.bbac409~ (2022)
- Zagirova D, "Biomedical generative pre-trained based transformer language model for age-related disease target discovery", Aging(Albany NY), 15, pp.9293~9309 (2023)