ChatGPT (GPT-4) passed the Japanese National License Examination for Pharmacists in 2022, answering all items including those with diagrams: a descriptive study
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Purpose: The objective of this study was to assess the performance of ChatGPT (GPT-4) on all items, including those with diagrams, in theJapanese National License Examination for Pharmacists (JNLEP) and compare it with the previous GPT-3.5 model’s performance. Methods: The 107th JNLEP, conducted in 2022, with 344 items input into the GPT-4 model, was targeted for this study. Separately, 284 items,excluding those with diagrams, were entered into the GPT-3.5 model. The answers were categorized and analyzed to determine accuracy ratesbased on categories, subjects, and presence or absence of diagrams. The accuracy rates were compared to the main passing criteria (overall accuracy rate ≥62.9%). Results: The overall accuracy rate for all items in the 107th JNLEP in GPT-4 was 72.5%, successfully meeting all the passing criteria. For the setof items without diagrams, the accuracy rate was 80.0%, which was significantly higher than that of the GPT-3.5 model (43.5%). The GPT-4model demonstrated an accuracy rate of 36.1% for items that included diagrams. Conclusion: Advancements that allow GPT-4 to process images have made it possible for LLMs to answer all items in medical-related licenseexaminations. This study’s findings confirm that ChatGPT (GPT-4) possesses sufficient knowledge to meet the passing criteria.
Abstract
Purpose: The objective of this study was to assess the performance of ChatGPT (GPT-4) on all items, including those with diagrams, in theJapanese National License Examination for Pharmacists (JNLEP) and compare it with the previous GPT-3.5 model’s performance. Methods: The 107th JNLEP, conducted in 2022, with 344 items input into the GPT-4 model, was targeted for this study. Separately, 284 items,excluding those with diagrams, were entered into the GPT-3.5 model. The answers were categorized and analyzed to determine accuracy ratesbased on categories, subjects, and presence or absence of diagrams. The accuracy rates were compared to the main passing criteria (overall accuracy rate ≥62.9%). Results: The overall accuracy rate for all items in the 107th JNLEP in GPT-4 was 72.5%, successfully meeting all the passing criteria. For the setof items without diagrams, the accuracy rate was 80.0%, which was significantly higher than that of the GPT-3.5 model (43.5%). The GPT-4model demonstrated an accuracy rate of 36.1% for items that included diagrams. Conclusion: Advancements that allow GPT-4 to process images have made it possible for LLMs to answer all items in medical-related licenseexaminations. This study’s findings confirm that ChatGPT (GPT-4) possesses sufficient knowledge to meet the passing criteria.
참고문헌 (14)
- Gilson A, "How does ChatGPT perform on the United States Medical Licensing Examination(USMLE)? : the implications of large language models for medical education and knowledge assessment", JMIR Med Educ, 9, pp.e45312~ (2023)[DOI]
- Weng TL, "ChatGPT failed Taiwan’s Family Medicine Board Exam", J Chin Med Assoc, 86, pp.762~766 (2023)[DOI]
- Shang L, "Can ChatGPT pass China’s national medical licensing examination?", Asian J Surg, 46, pp.6112~6113 (2023)[DOI]
- Flores-Cohaila JA, "Performance of ChatGPT on the Peruvian National Licensing Medical Examination : cross-sectional study", JMIR Med Educ, 9, pp.e48039~ (2023)[DOI]
- Wang YM, "Performance of ChatGPT on the pharmacist licensing examination in Taiwan", J Chin Med Assoc, 86, pp.653~658 (2023)[DOI]
- Kaneda Y, "Assessing the performance of GPT-3.5 and GPT-4 on the 2023 Japanese Nursing Examination", Cureus, 15, pp.e42924~ (2023)[DOI]
- Yanagita Y, "Accuracy of ChatGPT on Medical Questions in the National Medical Licensing Examination in Japan : evaluation study", JMIR Form Res, 7, pp.e48023~ (2023)[DOI]
- Takagi S, "Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: comparison study", JMIR Med Educ, 9, pp.e48002~ (2023)[DOI]
- Kunitsu Y, "The potential of GPT-4 as a support tool for pharmacists : analytical study using the Japanese National Examination for Pharmacists", JMIR Med Educ, 9, pp.e48452~ (2023)[DOI]
- "Question criteria for the Japanese national pharmacist examination" (2016)[URL]
- "Is the passing line for the 107th national pharmacist exam the same as last time?" (2022)[URL]
- Tanaka Y, "Performance of generative pretrained transformer on the National Medical Licensing Examination in Japan", PLOS Digit Health, 3, pp.e0000433~ (2024)[DOI]
- "Can ChatGPT pass the 2023 Japanese National Medical Licensing Examination?" (2023)[URL]
- Cohen A, "Performance of ChatGPT in Israeli Hebrew OBGYN national residency examinations", Arch Gynecol Obstet, 308, pp.1797~1802 (2023)[DOI]