Efficacy and limitations of ChatGPT as a biostatistical problem-solving tool in medical education in Serbia: a descriptive study
2
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
Purpose This study aimed to assess the performance of ChatGPT (GPT-3.5 and GPT-4) as a study tool in solving biostatistical problems and to identify any potential drawbacks that might arise from using ChatGPT in medical education, particularly in solving practical biostatistical problems. Methods ChatGPT was tested to evaluate its ability to solve biostatistical problems from the Handbook of Medical Statistics by Peacock and Peacock in this descriptive study. Tables from the problems were transformed into textual questions. Ten biostatistical problems were randomly chosen and used as text-based input for conversation with ChatGPT (versions 3.5 and 4). Results GPT-3.5 solved 5 practical problems in the first attempt, related to categorical data, cross-sectional study, measuring reliability, probability properties, and the t-test. GPT-3.5 failed to provide correct answers regarding analysis of variance, the chi-square test, and sample size within 3 attempts. GPT-4 also solved a task related to the confidence interval in the first attempt and solved all questions within 3 attempts, with precise guidance and monitoring. Conclusion The assessment of both versions of ChatGPT performance in 10 biostatistical problems revealed that GPT-3.5 and 4’s performance was below average, with correct response rates of 5 and 6 out of 10 on the first attempt. GPT-4 succeeded in providing all correct answers within 3 attempts. These findings indicate that students must be aware that this tool, even when providing and calculating different statistical analyses, can be wrong, and they should be aware of ChatGPT’s limitations and be careful when incorporating this model into medical education.
Abstract
Purpose This study aimed to assess the performance of ChatGPT (GPT-3.5 and GPT-4) as a study tool in solving biostatistical problems and to identify any potential drawbacks that might arise from using ChatGPT in medical education, particularly in solving practical biostatistical problems. Methods ChatGPT was tested to evaluate its ability to solve biostatistical problems from the Handbook of Medical Statistics by Peacock and Peacock in this descriptive study. Tables from the problems were transformed into textual questions. Ten biostatistical problems were randomly chosen and used as text-based input for conversation with ChatGPT (versions 3.5 and 4). Results GPT-3.5 solved 5 practical problems in the first attempt, related to categorical data, cross-sectional study, measuring reliability, probability properties, and the t-test. GPT-3.5 failed to provide correct answers regarding analysis of variance, the chi-square test, and sample size within 3 attempts. GPT-4 also solved a task related to the confidence interval in the first attempt and solved all questions within 3 attempts, with precise guidance and monitoring. Conclusion The assessment of both versions of ChatGPT performance in 10 biostatistical problems revealed that GPT-3.5 and 4’s performance was below average, with correct response rates of 5 and 6 out of 10 on the first attempt. GPT-4 succeeded in providing all correct answers within 3 attempts. These findings indicate that students must be aware that this tool, even when providing and calculating different statistical analyses, can be wrong, and they should be aware of ChatGPT’s limitations and be careful when incorporating this model into medical education.
참고문헌 (15)
- Ibrahim H, "Perception, performance, and detectability of conversational artificial intelligence across 32 university courses", Sci Rep, 13, pp.12187~ (2023)[DOI]
- Cheung BH, "ChatGPT versus human in generating medical graduate exam multiple choice questions: a multinational prospective study (Hong Kong S.A.R., Singapore, Ireland, and the United Kingdom)", PLoS One, 18, pp.e0290691~ (2023)[DOI]
- Moritz S, "Generative AI(gAI)in medical education : Chat-GPT and co", GMS J Med Educ (2023)[DOI]
- Kung TH, "Performance of ChatGPT on USMLE : potential for AI-assisted medical education using large language models", PLOS Digit Health, 2, pp.e0000198~ (2023)[DOI]
- Gilson A, "How does ChatGPT perform on the United States Medical Licensing Examination? : the implications of large language models for medical education and knowledge assessment", JMIR Med Educ, 9, pp.e45312~ (2023)[DOI]
- Chao-Cheng Lin, "Comparing ChatGPT’s ability to rate the degree of stereotypes and the consistency of stereotype attribution with those of medical students in New Zealand in developing a similarity rating test: a methodological study", Journal of Educational Evaluation for Health Professions, 20, pp.17~ (2023), 한국보건의료인국가시험원
- 허선, "Are ChatGPT’s knowledge and interpretation ability comparable to those of medical students in Korea for taking a parasitology examination?: a descriptive study", Journal of Educational Evaluation for Health Professions, 20, pp.1~ (2023), 한국보건의료인국가시험원
- AL-Qadri AH, "Assessing the ChatGPT accuracy through principles of statistics exam : a performance and implications", Res Sq (2023)[DOI]
- Dao XQ, "Investigating the effectiveness of ChatGPT in mathematical reasoning and problem solving : evidence from the Vietnamese National High School Graduation Examination", arXiv, ~ (2023)
- Peacock J, "Oxford handbook of medical statistics" (2011), Oxford University Press
- Shoufan A, "Exploring students’ perceptions of CHATGPT : thematic analysis and follow-up survey", IEEE Access, 11, pp.38805~38818 (2023)[DOI]
- Stanisavljevic D, "Assessing attitudes towards statistics among medical students : psychometric properties of the Serbian version of the Survey of Attitudes Towards Statistics(SATS)", PLoS One, 9, pp.e112567~ (2014)[DOI]
- Talan T, "The role of artificial intelligence in higher education : ChatGPT assessment for anatomy course", Uluslararası Yönetim Bilişim Sist Bilgi Bilim Derg, 7, pp.33~40 (2023)[DOI]
- Guo B, "How close is chatgpt to human experts? comparison corpus, evaluation, and detection", arXiv, ~ (2023)
- Hassani H, "The role of ChatGPT in data science : how ai-assisted conversational interfaces are revolutionizing the field", Big Data Cogn Comput, 7, pp.62~ (2023)[DOI]