Performance of ChatGPT on the Korean National Examination for Dental Hygienists
2
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Background: This study aimed to evaluate ChatGPT’s performance accuracy in responding to questions from the national dental hygienist examination. Moreover, through an analysis of ChatGPT’s incorrect responses, this research intended to pinpoint the predominant types of errors.Methods: To evaluate ChatGPT-3.5’s performance according to the type of national examination questions, the researchers classified 200 questions of the 49th National Dental Hygienist Examination into recall, interpretation, and solving type questions. The researchers strategically modified the questions to counteract potential misunderstandings from implied meanings or technical terminology in Korea. To assess ChatGPT-3.5’s problem-solving capabilities in applying previously acquired knowledge, the questions were first converted to subjective type. If ChatGPT-3.5 generated an incorrect response, an original multiple-choice framework was provided again. Two hundred questions were input into ChatGPT-3.5 and the generated responses were analyzed. After using ChatGPT, the accuracy of each response was evaluated by researchers according to the types of questions, and the types of incorrect responses were categorized (logical, information, and statistical errors). Finally, hallucination was evaluated when ChatGPT provided misleading information by answering something that was not true as if it were true. Results: ChatGPT’s responses to the national examination were 45.5% accurate. Accuracy by question type was 60.3% for recall and 13.0% for problem-solving type questions. The accuracy rate for the subjective solving questions was 13.0%, while the accuracy for the objective questions increased to 43.5%. The most common types of incorrect responses were logical errors 65.1% of all. Of the total 102 incorrectly answered questions, 100 were categorized as hallucinations. Conclusion: ChatGPT-3.5 was found to be limited in its ability to provide evidence-based correct responses to the Korean national dental hygiene examination. Therefore, dental hygienists in the education or clinical fields should be careful to use artificial intelligence-generated materials with a critical view.
Abstract
Background: This study aimed to evaluate ChatGPT’s performance accuracy in responding to questions from the national dental hygienist examination. Moreover, through an analysis of ChatGPT’s incorrect responses, this research intended to pinpoint the predominant types of errors.Methods: To evaluate ChatGPT-3.5’s performance according to the type of national examination questions, the researchers classified 200 questions of the 49th National Dental Hygienist Examination into recall, interpretation, and solving type questions. The researchers strategically modified the questions to counteract potential misunderstandings from implied meanings or technical terminology in Korea. To assess ChatGPT-3.5’s problem-solving capabilities in applying previously acquired knowledge, the questions were first converted to subjective type. If ChatGPT-3.5 generated an incorrect response, an original multiple-choice framework was provided again. Two hundred questions were input into ChatGPT-3.5 and the generated responses were analyzed. After using ChatGPT, the accuracy of each response was evaluated by researchers according to the types of questions, and the types of incorrect responses were categorized (logical, information, and statistical errors). Finally, hallucination was evaluated when ChatGPT provided misleading information by answering something that was not true as if it were true. Results: ChatGPT’s responses to the national examination were 45.5% accurate. Accuracy by question type was 60.3% for recall and 13.0% for problem-solving type questions. The accuracy rate for the subjective solving questions was 13.0%, while the accuracy for the objective questions increased to 43.5%. The most common types of incorrect responses were logical errors 65.1% of all. Of the total 102 incorrectly answered questions, 100 were categorized as hallucinations. Conclusion: ChatGPT-3.5 was found to be limited in its ability to provide evidence-based correct responses to the Korean national dental hygiene examination. Therefore, dental hygienists in the education or clinical fields should be careful to use artificial intelligence-generated materials with a critical view.
참고문헌 (15)
- Kung TH, "Performance of ChatGPT on USMLE : potential for AI-assisted medical education using large language models", PLOS Digit Health, 2, pp.e0000198~ (2023)[DOI]
- Gilson A, "How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? The implications of large language models for medical education and knowledge assessment", JMIR Med Educ, 9, pp.e45312~ (2023)[DOI]
- Brown TB, "Language models are few-shot learners", 34th International Conference on Neural Information Processing Systems (2020), Curran Associates Inc
- "What is ChatGPT: definition, how to use it" (2023)[URL]
- Sarraju A, "Appropriateness of cardiovascular disease prevention recommendations obtained from a popular online chatbased artificial intelligence model", JAMA, 329, pp.842~844 (2023)[DOI]
- Savery M, "Question-driven summarization of answers to consumer health questions", Sci Data, 7, pp.322~ (2020)[DOI]
- Das A, "Proceedings of the 21st Workshop on Biomedical Language Processing", pp.285~297 (2022), Association for Computational Linguistics
- Wójcik S, "Reshaping medical education:performance of ChatGPT on a PES medical examination", Cardiol J (2023)[DOI]
- Huh S, "Are ChatGPT’s knowledge and interpretation ability comparable to those of medical students in Korea for taking a parasitology examination? : a descriptive study", J Educ Eval Health Prof, 20, pp.1~ (2023)[DOI]
- Shin DK, "Exploring the potential of using ChatGPT as a content-based English learning and teaching tool", J Korea Eng Educ Soc, 22, pp.171~192 (2023)[DOI]
- Kim SH, "Dental hygienist job analysis for item development for the Korean dental hygienists’licensing examination", J Educ Eval Health Prof, 2, pp.59~74 (2005)[DOI]
- Kim SM, "Evaluation of the applicability of ChatGPT in biological nursing science education", J Korean Biol Nurs Sci, 25, pp.183~204 (2023)[DOI]
- Analysis report of the 49th national dental hygienist examination in 2021, "Kim JG", Korea Health Personnel Licensing Examination Institute, 1~33 (2022)
- Kim SW, "Evaluating Korean language capability of ChatGPT and discussion", Korean Inst Inf Sci Eng Proc Korea Comput Congr, 2023, pp.286~288 (2023)
- Sun GH, "The ChatGPT storm and what faculty can do", Nurse Educ, 48, pp.119~124 (2023)[DOI]