Examining AI-Based Accuracy Assessment in L2 Learners’ Writing
2
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Despite the increasing interest in using AI tools as assistant agents in instructional settings, the effectiveness of ChatGPT, the generative pre trained AI, for evaluating the accuracy of second language (L2) writing has been largely unexplored in formative assessment. Therefore, the current study aims to examine how ChatGPT, as an evaluator, performs in evaluating the accuracy of L2 writing, compared to human evaluators. Thirty-one academic writings were assessed by two human raters and ChatGPT. Both human and AI raters evaluated L2 learners’writings based on holistic measures (i.e., IELTS academic writing rubrics), focusing on coherence, lexical resources, grammatical accuracy, and global and specific measures of accuracy, by counting grammatical and lexical errors. The results revealed a moderate positive correlation in scores from holistic measures by humans and ChatGPT. This finding suggests that AI tools have the potential to assist both human raters and EFL learners in evaluating L2 writing, thereby highlighting the effective potential of employing AI tools as pedagogical aids within educational settings.
Abstract
Despite the increasing interest in using AI tools as assistant agents in instructional settings, the effectiveness of ChatGPT, the generative pre trained AI, for evaluating the accuracy of second language (L2) writing has been largely unexplored in formative assessment. Therefore, the current study aims to examine how ChatGPT, as an evaluator, performs in evaluating the accuracy of L2 writing, compared to human evaluators. Thirty-one academic writings were assessed by two human raters and ChatGPT. Both human and AI raters evaluated L2 learners’writings based on holistic measures (i.e., IELTS academic writing rubrics), focusing on coherence, lexical resources, grammatical accuracy, and global and specific measures of accuracy, by counting grammatical and lexical errors. The results revealed a moderate positive correlation in scores from holistic measures by humans and ChatGPT. This finding suggests that AI tools have the potential to assist both human raters and EFL learners in evaluating L2 writing, thereby highlighting the effective potential of employing AI tools as pedagogical aids within educational settings.
참고문헌 (44)
- Ahmadian, M. J., "The effects of simultaneous use of careful online planning and task repetition on accuracy, complexity, and fluency in EFL learners’ oral production", Language Teaching Research, 15(1), pp.35~59 (2011)
- Bai, L., "In the face of fallible AWE feedback: How do students respond?", Educational Psychology, 37(1), pp.67~81 (2017)
- Barrot, J., "Using ChatGPT for second language writing : Pitfalls and potentials", Assessing Writing, 57, pp.100745~ (2023)
- Bridgemand. B., "Design and evaluation of automated writing evaluation models : Relationships with writing in naturalistic setting", Assessing Writing, 34(3), pp.62~71 (2017)
- Burstein, J., "Informing automated writing evaluation using the lens of genre : Two studies", CALICO Journal, 33(1), pp.117~141 (2016)
- Bygate, M., "Researching pedagogic tasks: Second language learning and testing", pp.23~48 (2001), Longman
- Camacho, A., "Writing motivation in school : A systematic review of empirical research in the early twenty-first century", Educational Psychology Review, 33(1), pp.213~247 (2020)
- Ding, L., "Automated writing evaluation systems: A systematic review of Grammarly, Pigai, and Criterion with a –perspective on future directions in the age of generative artificial intelligence", Education and Information Technologies, 29(11), pp.1~53 (2024)
- Evans, N. W., "Contextualizing corrective feedback in second language writing pedagogy", Language Teaching Research, 14(4), pp.445~463 (2010)
- Ferris, D., "Error feedback in L2 writing classes: How explicit does it need to be?", Journal of Second Language Writing, 10(3), pp.161~184 (2001)
- Foster, P., "The influence of planning and task type on second language performance", Studies in Second Language Acquisition, 18(3), pp.299~323 (1996)
- Godwin-Jones, R., "Big data and language learning opportunities and challenges", Language Learning & Technology, 25(1), pp.4~19 (2021)
- Grimes, D., "Utility in a fallible tool: A multi-site case study of automated writing evaluation", The Journal of Technology, Learning, and Assessment, 8(6), pp.1~43 (2010)
- Haugnes, N., "North star 2: Reading & writing level 2" (2017), Pearson Education
- Hibert, A. I., "Systematic literature review of automated writing evaluations as formative learning tool", The proceedings of transforming learning with meaningful technologies, pp.199~212 (2017), Springer
- Huang, Y., "Using automated feedback to develop writing proficiency", Computers and Composition, 62, pp.102675~ (2021)
- Hussein, M., "Automated language essay scoring systems: A literature review", Peer Computer Science, 5, pp.e208~ (2019)
- Hwang, K. -H., "The implementation of ChatGPT-assisted writing instruction in ESL", EFL classrooms, 2023, pp.65(3), 83~106
- John, P., "Using grammar checkers in an ESL context : An investigation of automatic corrective feedback", CALICO Journal, 37(2), pp.193~196 (2020)
- Lee, H., "A study of evaluation and feedback analysis of Korean English instructors, native English instructors, and ChatGPT on college student’ English writing", The Journal of Modern British & American Language & Literature, 41(4), pp.247~270 (2023)
- Lee, O. -S., "Re-examining accuracy measures in task-based research: Task repetition", Journal of Research in Curriculum and Instruction, 25(5), pp.315~325 (2021)
- Link, S., "Impact of automated writing evaluation on teacher feedback, student revision, and writing improvement", Computer Assisted Language Learning, 35(4), pp.605~634 (2022)
- Mehnert, U., "The effects of different lengths of time for planning on second language performance", Studies in Second Language Acquisition, 20(1), pp.83~108 (1998)
- Mizumoto, A., "Testing the viability of ChatGPT as a companion in L2 writing accuracy assessment", Research Methods in Applied Linguistics, 3, pp.100~116 (2024)
- Nunes, A., "Effectiveness of automated writing evaluation systems in school setting : A systematic review of studies from 2000 to 2020", Journal of Computer Assisted Learning, 38(2), pp.599~620 (2021)
- Ojima, M., "Concept mapping as pre-task planning : A case study of three Japanese ESL writers", System, 34, pp.566~585 (2006)
- Ortega, L., "New CALL-SAL research interfaces for the 21st century : Towards equitable multilingualism", CALICO Journal, 34(3), pp.283~316 (2017)
- Park, H. -k., "The use of ChatGPT for grammatical error analysis in EFL writing : A case of Korean university students", Studies in English Education, 28(4), pp.25~53 (2023)
- Park, H., "ChatGPT-integrated English writing : Writing strategies and perceptions", The Linguistic Association of Korea Journal, 32(3), pp.85~103 (2024)
- Polio, C., "Measure of linguistic accuracy in second language writing research", Language Learning, 47(1), pp.101~143 (1997)
- Polio, C., "An investigation into current measures of linguistic accuracy in second language writing research", Journal of Second Language Writing, 26(1), pp.10~27 (2014)
- Song, C., "Enhancing academic writing skills and motivation: Assessing the efficacy of Chat GPT in AI-assisted language learning for EFL students", Frontiers in Psychology, 14, pp.1260843~ (2023)
- Ranalli, J., "Automated written corrective feedback : Error-correction performance and timing of delivery", Language Learning & Technology, 26(1), pp.1~25 (2022)
- Ranalli, J., "Automated written corrective feedback: How well can students make use of it?", Computer Assisted Language Learning, 31(7), pp.653~674 (2018)
- Ranalli, J., "L2 student engagement with automated feedback on writing:Potential for learning and issues of trust", Journal of Second Language Writing, 52, pp.100816~ (2021)
- Ranalli, J., "Automated writing evaluation for formative assessment of second language writing : Investigating the accuracy and usefulness of feedback as part of argument-based validation", Educational Psychology, 37(1), pp.8~25 (2017)
- Shi, H., "A systematic review of AI-based automated written feedback research", ReCALL, 36(2), pp.187~209 (2024)
- Skehan, P., "The influence of task structure and processing conditions on narrative retellings", Language Learning, 49(1), pp.93~120 (1999)
- Thewissen, J., "Capturing L2 accuracy developmental patterns : Insights from an error-tagged EFL learner corpus", The Modern Language Journal, 97(1), pp.77~101 (2013)
- Thorp, H., "ChatGPT is fun, but not an author", Science, 379(6630), pp.313~ (2023)
- Wolfe-Quintero, K., "Second language development in writing: Measures of fluency, accuracy and complexity" (1998), University of Hawai’i Press
- Warschauer, M., "Automated writing evaluation : Defining the classroom research agenda", Language Teaching Research, 10(2), pp.157~180 (2006)
