Performance of ChatGPT 3.5 on CSAT: Its Potential as a Language Learning and Assessment Tool
0
조회
0
다운로드
11
인용
0
참고문헌
키워드
초록
The purpose of this paper is to evaluate ChatGPT 3.5’s performance on the English reading comprehension section of the Korean College Scholastic Ability Test (CSAT). More specifically, this study analyzes its potential as a language learning and assessment tool based on its performance on the high-stakes national test. This study also identifies the strengths and limitations of the large language model (LLM) and explores aspects that need improvement to use for learning. The results suggest that the LLM’s strengths in reasoning and explanation can be useful for language learning. The results also highlight the need for ChatGPT to gain extensive domain-specific knowledge and advanced reasoning skills in order to enhance its contextual understanding. Moreover, the results indicate that human judgment from educators is necessary to ensure the accuracy and reliability of ChatGPT’s outputs, particularly for a high-stakes exam like the CSAT. Combining AI and human judgment can result in more effective and trustworthy results.
Abstract
The purpose of this paper is to evaluate ChatGPT 3.5’s performance on the English reading comprehension section of the Korean College Scholastic Ability Test (CSAT). More specifically, this study analyzes its potential as a language learning and assessment tool based on its performance on the high-stakes national test. This study also identifies the strengths and limitations of the large language model (LLM) and explores aspects that need improvement to use for learning. The results suggest that the LLM’s strengths in reasoning and explanation can be useful for language learning. The results also highlight the need for ChatGPT to gain extensive domain-specific knowledge and advanced reasoning skills in order to enhance its contextual understanding. Moreover, the results indicate that human judgment from educators is necessary to ensure the accuracy and reliability of ChatGPT’s outputs, particularly for a high-stakes exam like the CSAT. Combining AI and human judgment can result in more effective and trustworthy results.
참고문헌 (21)
- "Advantages and disadvantages of deep learning" (2022)[URL]
- Michael James Bommarito, "GPT Takes the Bar Exam", SSRN Electronic Journal (2022), Elsevier BV[DOI]
- 장지연, "평가 유형(상대평가 대 절대평가)에 따른 2016-2019 수능 영어 독해 지문의 어휘 복잡성 비교", The SNU Journal of Education Research, 28(3), pp.85~110 (2019), 교육종합연구원[DOI]
- 최민주, "수능 영어 문항 유형간 응집력과 어휘정보 분석", 한국콘텐츠학회 논문지, 17(12), pp.378~385 (2017), 한국콘텐츠학회[DOI]
- Gilson, A., "How does ChatGPT perform on the medical licensing exams? The implications of large language models for medical education and knowledge assessment", JMIR Medical Education (2022)
- 황이수, "수학능력시험 영어 지문의 텍스트 요인과 문항 난이도의 상관관계 분석: 통사적 복잡성을 중심으로", 영어영문학연구, 46(1), pp.265~283 (2020), 대한영어영문학회[DOI]
- 황이수, "수학능력시험 영어 영역 읽기 지문의 이독성 및 어휘 정보와 문항 난이도 간의 상관관계", 인문사회 21, 11(2), pp.389~400 (2020), 인문사회 21
- "Advantages and disadvantages of deep learning" (2023)[URL]
- "Previous college scholastic ability test exams" (2022)[URL]
- "KICE", 2016
- Kung, T, H., "Performance of ChatGPT on USMLE : Potential for AI-assisted medical education using large language models", PLOS Digital Health, 2(2) (2023)
- 권오량, "대학수학능력시험 외국어(영어) 영역 정책 변천사", 영어교육, 70, pp.3~34 (2015), 한국영어교육학회[DOI]
- 권서경, "대학수학능력시험 영어영역 읽기 문항 난이도 요인 탐색: EBS 연계정책 변화를 중심으로", 영어교과교육, 21(2), pp.49~71 (2022), 한국영어교과교육학회
- 권서경, "EBS 연계정책에 따른 대학수학능력시험 영어영역 어휘 난이도 변화 분석", 영어교과교육, 13(4), pp.97~121 (2014), 한국영어교과교육학회[DOI]
- 이현우, "절대평가 시험으로서의 수능 영어시험 분석: 주요 난이도 예측 변인을 중심으로", 중등영어교육, 14(4), pp.90~124 (2021), 한국중등영어교육학회[DOI]
- 이정화, "Human Interpretation and Machine Translations Based upon Interviews with Director Joon-ho Bong", 영어학, 23, pp.204~219 (2023), 한국영어학회[DOI]
- "GPT-4 technical report" (2023)[URL]
- "The questions that determined the score of CSAT with a 0% answer rate.. Statistics 29, Geometry 30" (2022)[URL]
- Would ChatGPT3 get a Wharton MBA? A prediction based on its performance in the operations management course, "Terwiesch, C.", Mack Institute for Innovation Management at the Wharton School, ~ (2023)
- 우진하, "Systematic Review for AI-based Language Learning Tools", 디지털콘텐츠학회논문지, 22(11), pp.1783~1792 (2021), 한국디지털콘텐츠학회[DOI]
- 우진하, "Utilizing Machine Translation Systems to Generate Word Lists for Learning Vocabulary in English", 디지털콘텐츠학회논문지, 22(1), pp.71~75 (2021), 한국디지털콘텐츠학회[DOI]