ChatGPT와 코퍼스 기반의 대학수학능력시험 모의평가 영어영역 실제 정답률 예측 가능성 모색
Predicting the difficulty of college scholastic ability test based on ChatGPT and corpus linguistic features
0
조회
0
다운로드
2
인용
0
참고문헌
키워드
초록
This study attempts to validate ChatGPT, as well as corpus linguistic features, in predicting the difficulty level of the College Scholastic Ability Test(CSAT). The Observed Proportion Correct (Pc) of the 2021 to 2024 CSAT mock tests was compared with those provided by ChatGPT-3.5, ChatGPT-4.0 and 10 corpus-based linguistic features, including Type, Token, TTR, AWL, Lexical Density, Average Words per Sentence, Flesch Reading Ease, Flesch-Kincaid Grade Level, and Gunning-Fog Index. The research findings are as follows: First, ChatGPT-4.0 bettter predicted item difficulty compared to ChatGPT-3.5. Second, the Predicted Proportion Correct (Pc) of ChatGPT-3.5, ChatGPT-4.0 and some corpus features showed a high correlation with observed Pc. The observed Pc was largely accounted for by the difficulty predicted by ChatGPT-3.5 and ChatGPT-4.0, where ChatGPT-4.0’s predicted Pc demonstrated higher explanatory power than ChatGPT-3.5. Third, a regression model for predicting the difficulty of CSAT was established with ChatGPT and such corpus-based variables as the Gunning-Fog Index and Lexical Diversity. Overall, the findings endorse the potential use of ChatGPT, along with salient corpus linguistic features, as a suitable tool for predicting item difficulties prior to actual test administration.
Abstract
This study attempts to validate ChatGPT, as well as corpus linguistic features, in predicting the difficulty level of the College Scholastic Ability Test(CSAT). The Observed Proportion Correct (Pc) of the 2021 to 2024 CSAT mock tests was compared with those provided by ChatGPT-3.5, ChatGPT-4.0 and 10 corpus-based linguistic features, including Type, Token, TTR, AWL, Lexical Density, Average Words per Sentence, Flesch Reading Ease, Flesch-Kincaid Grade Level, and Gunning-Fog Index. The research findings are as follows: First, ChatGPT-4.0 bettter predicted item difficulty compared to ChatGPT-3.5. Second, the Predicted Proportion Correct (Pc) of ChatGPT-3.5, ChatGPT-4.0 and some corpus features showed a high correlation with observed Pc. The observed Pc was largely accounted for by the difficulty predicted by ChatGPT-3.5 and ChatGPT-4.0, where ChatGPT-4.0’s predicted Pc demonstrated higher explanatory power than ChatGPT-3.5. Third, a regression model for predicting the difficulty of CSAT was established with ChatGPT and such corpus-based variables as the Gunning-Fog Index and Lexical Diversity. Overall, the findings endorse the potential use of ChatGPT, along with salient corpus linguistic features, as a suitable tool for predicting item difficulties prior to actual test administration.
참고문헌 (37)
- 안유영, "Performance of ChatGPT 3.5 on CSAT: Its Potential as a Language Learning and Assessment Tool", 영어교과교육, 22(2), pp.119~145 (2023), 한국영어교과교육학회
- Bachman, L. F., "An investigation into the comparability of two tests of English as a foreign langauge" (1995), Cambridge University Press
- 백미경, "ChatGPT의 한국어 능력 평가에 대한 연구 - 한국어능력시험 읽기 영역 및 관용구를 중심으로 -", 언어사실과 관점, 59, pp.279~308 (2023), 언어정보연구원[DOI]
- "Text Inspector: Online text analysis tool" (2012)[URL]
- Caramancion, K. M., "News verifiers showdown: A comparative performance evaluation of chatgpt 3.5, chatgpt 4.0, bing ai, and bard in news fact-checking", ~ (2023)
- Choi, I. -C., "Predicting the difficulty of EFL tests based on corpus linguistic features and expert judgment", Language Assessment Quarterly, 17(1), pp.18~42 (2020)[DOI]
- 최윤하, "CEFR 기반 2015 개정 교육과정, 교과서, 수능의 일관성: 영어 읽기를 중심으로", 영어교육, 78(1), pp.271~291 (2023), 한국영어교육학회[DOI]
- "Web Vocabprofile: An adaptation of Heatley, Nation, & Coxhead's" (2002)[URL]
- Coxhead, A., "A new academic word list", TESOL Quarterly, 34(2), pp.213~238 (2000)
- de Winter, J. C., "Can chatgpt pass high school exams on English language comprehension?", International Journal of Artificial Intelligence in Education (2023)[DOI]
- Freedle, R., "The prediction of TOEFL reading item difficulty : Implications for construct validity", Language Testing, 10(2), pp.133~170 (1993)[DOI]
- Gunning, R, "The technique of clear writing" (1952), McGraw-Hill
- Hamada, A., "Linguistic variables determining the difficulty of Eiken reading passages", JLTA Journal, 18, pp.57~77 (2015)[DOI]
- Jeon, J., "Large language models in education: A focus on the complementary relationship between human teachers and ChatGPT", Education and Information Technologies, 28, pp.15873~15892 (2023)[DOI]
- Kim, J., "A diachronic analysis of English KSAT reading passages", Journal of the Elementary Society, 27, pp.63~78 (2017)
- 김재은, "고등학교 영어 교과서, EBS 수능 연계 교재, 대학수학능력시험의 코퍼스기반 난이도 비교 분석", 멀티미디어 언어교육, 18(1), pp.59~92 (2015), 한국멀티미디어언어교육학회[DOI]
- 김태국, "챗GPT를 활용한 중등 영어 교수 자료 개발 연구", 중등영어교육, 16(2), pp.207~218 (2023), 한국중등영어교육학회
- Derivation of new readability formulas for Navy enlisted personnel, "Kincaid, J. P.", Naval Technical Training, ~ (1975)
- College scholastic ability test Q&A, "Korea Institute for Curriculum and Evaluation", Korea Institute for Curriculum and Evaluation, ~ (2001)
- "Press release on the scoring results of the 2021 college scholastic ability test" (2020)[URL]
- "Press release on the scoring results of the 2022 college scholastic ability test" (2021)[URL]
- "Press release on the scoring results of the 2023 college scholastic ability test" (2022)[URL]
- 권서경, "대학수학능력시험 영어영역 읽기 문항 난이도 요인 탐색: EBS 연계정책 변화를 중심으로", 영어교과교육, 21(2), pp.49~71 (2022), 한국영어교과교육학회
- Liebrenz, M., "Generating scholarly content with ChatGPT: Ethical challenges for medical publishing", The Lancet Digital Health, 5(3), pp.105~106 (2023)[DOI]
- Loukina, A., "Textual complexity as a predictor of difficulty of listening items in language proficiency tests", Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp.3245~3253 (2016)
- "Introduction of absolute evaluation in the English section of the College Scholastic Ability Test: To be applied from the 2018 academic year for students who are currently in the third year of middle school" (2014)[URL]
- 문지현, "대학수학능력시험 영어 읽기 지문의 언어적 요소 분석", 현대영어교육, 18(1), pp.193~211 (2017), 현대영어교육학회
- 오준일, "수능 영어 독해 지문의 언어적 복잡성에 관한 연구", 영어교육연구, 32(2), pp.109~126 (2020), 팬코리아영어교육학회[DOI]
- 엄혜랑, "절대 평가제 도입 이후 수능 영어 독해지문의 난이도 분석: 어휘 복잡성과 통사적 복잡성을 중심으로", 학습자중심교과교육연구, 22(6), pp.397~416 (2022), 학습자중심교과교육학회[DOI]
- Om, H., "Validation of the test usefulness of the Korean college transfer exam of English", English Language Assessment, 10(1), pp.61~96 (2015)
- "GPT-4 technical report" (2023)[URL]
- 박정곤, "Coh-Metrix를 활용한 고등학교 영어 교과서와 대학수학능력시험 읽기 지문 난이도 비교 연구", 영어영문학연구, 47(3), pp.147~170 (2021), 대한영어영문학회[DOI]
- Pavlik, J. V., "Collaborating with ChatGPT: Considering the implications of generative artificial intelligence for journalism and media education", Journalism & Mass Communication Educator, 78(1), pp.84~93 (2023)[DOI]
- Rupp, A. A., "Combining multiple regression and CART to understand difficulty in second language reading and listening comprehension test items", International Journal of Testing, pp.185~216 (2001)[DOI]
- 신유선, "Coh-Metrix와 VocabProfile를 활용한 수능 영어 독해 지문의 분석: 2016-2019년 4년을 중심으로", 언어과학, 26(4), pp.109~127 (2019), 한국언어과학회[DOI]
- Takagi, S., "Performance of GPT-3.5 and GPT-4 on the Japanese medical licensing examination: comparison study", JMIR Medical Education, 9, pp.e48002~ (2023)[DOI]
- Yan, D., "Impact of ChatGPT on learners in a L2 writing practicum: An exploratory investigation", Education and Information Technologies, 28(11), pp.1~25 (2023)[DOI]