Performance of ChatGPT, Bard, Claude, and Bing on the Peruvian National Licensing Medical Examination: a cross-sectional study
0
조회
0
다운로드
2
인용
0
참고문헌
키워드
초록
Purpose We aimed to describe the performance and evaluate the educational value of justifications provided by artificial intelligence chatbots, including GPT-3.5, GPT-4, Bard, Claude, and Bing, on the Peruvian National Medical Licensing Examination (P-NLME). Methods This was a cross-sectional analytical study. On July 25, 2023, each multiple-choice question (MCQ) from the P-NLME was entered into each chatbot (GPT-3, GPT-4, Bing, Bard, and Claude) 3 times. Then, 4 medical educators categorized the MCQs in terms of medical area, item type, and whether the MCQ required Peru-specific knowledge. They assessed the educational value of the justifications from the 2 top performers (GPT-4 and Bing). Results GPT-4 scored 86.7% and Bing scored 82.2%, followed by Bard and Claude, and the historical performance of Peruvian examinees was 55%. Among the factors associated with correct answers, only MCQs that required Peru-specific knowledge had lower odds (odds ratio, 0.23; 95% confidence interval, 0.09–0.61), whereas the remaining factors showed no associations. In assessing the educational value of justifications provided by GPT-4 and Bing, neither showed any significant differences in certainty, usefulness, or potential use in the classroom. Conclusion Among chatbots, GPT-4 and Bing were the top performers, with Bing performing better at Peru-specific MCQs. Moreover, the educational value of justifications provided by the GPT-4 and Bing could be deemed appropriate. However, it is essential to start addressing the educational value of these chatbots, rather than merely their performance on examinations.
Abstract
Purpose We aimed to describe the performance and evaluate the educational value of justifications provided by artificial intelligence chatbots, including GPT-3.5, GPT-4, Bard, Claude, and Bing, on the Peruvian National Medical Licensing Examination (P-NLME). Methods This was a cross-sectional analytical study. On July 25, 2023, each multiple-choice question (MCQ) from the P-NLME was entered into each chatbot (GPT-3, GPT-4, Bing, Bard, and Claude) 3 times. Then, 4 medical educators categorized the MCQs in terms of medical area, item type, and whether the MCQ required Peru-specific knowledge. They assessed the educational value of the justifications from the 2 top performers (GPT-4 and Bing). Results GPT-4 scored 86.7% and Bing scored 82.2%, followed by Bard and Claude, and the historical performance of Peruvian examinees was 55%. Among the factors associated with correct answers, only MCQs that required Peru-specific knowledge had lower odds (odds ratio, 0.23; 95% confidence interval, 0.09–0.61), whereas the remaining factors showed no associations. In assessing the educational value of justifications provided by GPT-4 and Bing, neither showed any significant differences in certainty, usefulness, or potential use in the classroom. Conclusion Among chatbots, GPT-4 and Bing were the top performers, with Bing performing better at Peru-specific MCQs. Moreover, the educational value of justifications provided by the GPT-4 and Bing could be deemed appropriate. However, it is essential to start addressing the educational value of these chatbots, rather than merely their performance on examinations.
참고문헌 (13)
- Flores-Cohaila JA, "Performance of ChatGPT on the Peruvian National Licensing Medical Examination : cross-sectional study", JMIR Med Educ, 9, pp.e48039~ (2023)[DOI]
- Nori H, "Capabilities of GPT-4 on medical challenge problems", arXiv, ~ (2023)
- Wang H, "Performance and exploration of ChatGPT in medical examination, records and education in Chinese : pave the way for medical AI", Int J Med Inform, 177, pp.105173~ (2023)[DOI]
- Kumari A, "Large language models in hematology case solving: a comparative study of ChatGPT-3.5, Google Bard, and Microsoft Bing", Cureus, 15, pp.e43861~ (2023)[DOI]
- Mendoza-Chuctaya G, "Examen Nacional de Medicina (ENAM): analysis of the last decade of theoretical evaluations in future doctors in Peru", Acta Med Peru, 38, pp.169~176 (2021)[DOI]
- 허선, "Are ChatGPT’s knowledge and interpretation ability comparable to those of medical students in Korea for taking a parasitology examination?: a descriptive study", Journal of Educational Evaluation for Health Professions, 20, pp.1~ (2023), 한국보건의료인국가시험원
- Ediger D, "Academic Life in Emergency Medicine(ALiEM)Blog and Podcast Watch : infectious diseases", Cureus, 12, pp.e7674~ (2020)[DOI]
- "ENAM specifications table" (2020)[URL]
- Lai H, "Using automatic item generation to improve the quality of MCQ distractors", Teach Learn Med, 28, pp.166~173 (2016)[DOI]
- Chan TM, "Examining reliability and validity of an online score(ALiEM AIR)for rating free open access medical education resources", Ann Emerg Med, 68, pp.729~735 (2016)[DOI]
- McHugh ML, "Interrater reliability: the kappa statistic", Biochem Med(Zagreb), 22, pp.276~282 (2012)[DOI]
- Lee H, "The rise of ChatGPT : Exploring its potential in medical education", Anat Sci Educ (2023)[DOI]
- Takagi S, "Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: comparison study", JMIR Med Educ, 9, pp.e48002~ (2023)[DOI]