Evaluating the Accuracy of Artificial Intelligence-Based Chatbots on Pediatric Dentistry Questions in the Korean National Dental Board Exam
2
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
This study aimed to assess the competency of artificial intelligence (AI) in pediatric dentistry and compare it with that of dentists. We used open-source data obtained from the Korea Health Personnel Licensing Examination Institute. A total of 32 item multiple-choice pediatric dentistry exam questions were included. Two AIbased chatbots (ChatGPT 3.5 and Gemini) were evaluated. Each chatbot received the same questions seven times in separate chat sessions initiated on April 25, 2024. The accuracy was assessed by measuring the percentage of correct answers, and consistency was evaluated using Cronbach’s alpha coefficient. Both ChatGPT 3.5 and Gemini demonstrated similar accuracy, with no significant differences observed between them. However, neither chatbot achieved the minimum passing score set by the Pediatric Dentistry National Examination. However, both chatbots exhibited acceptable consistency in their responses. Within the limits of this study, both AIbased chatbots did not sufficiently answer the pediatric dentistry exam questions. This finding suggests that pediatric dentists should be aware of the advantages and limitations of this new tool and effectively utilize it to promote patient health.
Abstract
This study aimed to assess the competency of artificial intelligence (AI) in pediatric dentistry and compare it with that of dentists. We used open-source data obtained from the Korea Health Personnel Licensing Examination Institute. A total of 32 item multiple-choice pediatric dentistry exam questions were included. Two AIbased chatbots (ChatGPT 3.5 and Gemini) were evaluated. Each chatbot received the same questions seven times in separate chat sessions initiated on April 25, 2024. The accuracy was assessed by measuring the percentage of correct answers, and consistency was evaluated using Cronbach’s alpha coefficient. Both ChatGPT 3.5 and Gemini demonstrated similar accuracy, with no significant differences observed between them. However, neither chatbot achieved the minimum passing score set by the Pediatric Dentistry National Examination. However, both chatbots exhibited acceptable consistency in their responses. Within the limits of this study, both AIbased chatbots did not sufficiently answer the pediatric dentistry exam questions. This finding suggests that pediatric dentists should be aware of the advantages and limitations of this new tool and effectively utilize it to promote patient health.
참고문헌 (33)
- Arif TB, "The future of medical education and research : Is ChatGPT a blessing or blight in disguise?", Med Educ Online, 28, pp.2181052~ (2023)
- Manickam P, "Artificial Intelligence(AI)and Internet of Medical Things(IoMT)Assisted Biomedical Systems for Intelligent Healthcare", Biosensors(Basel), 12, pp.562~ (2022)
- Lee JW, "Development of AI-generated medical responses using the ChatGPT for cancer patients", Comput Methods Programs Biomed, 254, pp.108302~ (2024)
- Schwendicke F, "Artificial intelligence in dental research : Checklist for authors, reviewers, readers", J Dent, 107, pp.103610~ (2021)
- McDermott MBA, "Reproducibility in machine learning for health research: Still a ways to go", Sci Transl Med, 13, pp.eabb1655~ (2021)
- Vaishya R, "ChatGPT : Is this version good for healthcare and research?", Diabetes Metab Syndr, 17, pp.102744~ (2023)
- Gilson A, "How Does ChatGPT Perform on the United States Medical Licensing Examination(USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment", JMIR Med Educ, 9, pp.E45312~ (2023)
- Potapenko I, "Artificial intelligence-based chatbot patient information on common retinal diseases using ChatGPT", Acta Ophthalmol, 101, pp.829~831 (2023)
- Yeo YH, "Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma", Clin Mol Hepatol, 29, pp.721~732 (2023)
- Kung TH, "Performance of ChatGPT on USMLE : potential for AI-assisted medical education using large language models", PLoS Digit Health, 2, pp.E0000198~ (2023)
- Frosolini A, "Assessing the accuracy of ChatGPT references in head and neck and ENT disciplines", Eur Arch Otorhinolaryngol, 280, pp.5129~5133 (2023)
- Lum ZC, "Can Artificial Intelligence Pass the American Board of Orthopaedic Surgery Examination? Orthopaedic Residents Versus ChatGPT", Clin Orthop Relat Res, 481, pp.1623~1630 (2023)
- Schwendicke F, "Artificial Intelligence in Dentistry : Chances and Challenges", J Dent Res, 99, pp.769~774 (2020)
- Taber KS, "The use of Cronbach’s alpha when developing and reporting research instruments in science education", Res Sci Educ, 48, pp.1273~1296 (2018)
- "Results of the analysis of the 75th national examination of dentists in 2023"[URL]
- Strong E, "Chatbot vs Medical Student Performance on Free-Response Clinical Reasoning Examinations", JAMA Intern Med, 183, pp.1028~1030 (2023)
- Hoch CC, "ChatGPT's quiz skills in different otolaryngology subspecialties : an analysis of 2576 single-choice and multiple-choice board certification preparation questions", Eur Arch Otorhinolaryngol, 280, pp.4271~4278 (2023)
- Alhaidry HM, "ChatGPT in Dentistry : A Comprehensive Review", Cureus, 15, pp.E38317~ (2023)
- Liu J, "Utility of ChatGPT in Clinical Practice", J Med Internet Res, 25, pp.E48568~ (2023)
- Sallam M, "ChatGPT Utility in Healthcare Education, Research, and Practice : Systematic Review on the Promising Perspectives and Valid Concerns", Healthcare(Basel), 11, pp.887~ (2023)
- Haupt CE, "AI-generated medical advice-GPT and beyond", JAMA, 329, pp.1349~1350 (2023)
- Athaluri SA, "Exploring the Boundaries of Reality : Investigating the Phenomenon of Artificial Intelligence Hallucination in Scientific Writing Through ChatGPT References", Cureus, 15, pp.E37432~ (2023)
- Ramgopal S, "Artificial intelligence-based clinical decision support in pediatrics", Pediatr Res, 93, pp.334~341 (2023)
- Liu Z, "Surviving ChatGPT in healthcare", Front Radiol, 3, pp.1224682~ (2024)
- Dave T, "ChatGPT in medicine : an overview of its applications, advantages, limitations, future prospects, and ethical considerations", Front Artif Intell, 6, pp.1169595~ (2023)
- Stokel-Walker C, "ChatGPT listed as author on research papers : many scientists disapprove", Nature, 613, pp.620~621 (2023)
- Chatterjee J, "This new conversational AI model can be your friend, philosopher, and guide ... and even your worst enemy", Patterns (N Y), 4, pp.100676~ (2023)
- Shen Y, "ChatGPT and Other Large Language Models Are Double-edged Swords", Radiology, 307, pp.E230163~ (2023)
- Biswas S, "ChatGPT and the Future of Medical Writing", Radiology, 307, pp.E223312~ (2023)
- WHO, "WHO guideline Recommendations on Digital Interventions for Health System Strengthening" (2019), World Health Organization
- Balel Y, "Can ChatGPT be used in oral and maxillofacial surgery?", J Stomatol Oral Maxillofac Surg, 124, pp.101471~ (2023)
- Guerra GA, "GPT-4 Artificial Intelligence Model Outperforms ChatGPT, Medical Students, and Neurosurgery Residents on Neurosurgery Written Board-Like Questions", World Neurosurg, 179, pp.E160~E165 (2023)