LLM 기반 EFL 에세이 다면평가 프레임워크 개발 연구
A Study on the Development of an LLM-Based Multi-Trait Essay Scoring Framework for EFL Learners
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Essay scoring has garnered increasing attention in the field of EFL education due to its efficacy in assessing learners’ cognitive and communicative competencies. However, manually scoring essays often encounters significant bottlenecks, including a lack of scoring expertise, inconsistencies among raters, and substantial time demands. To tackle these challenges, this study proposes a Large Language Model-based Multi-Trait Essay Scoring (LMES) framework, which incorporates rubric-based criteria into the reasoning process of an LLM to generate textual justifications, referred to as rationales. These rationales subsequently guide a BERT language model in predicting trait-specific scores. Utilizing the Feedback Prize essay dataset, LMES demonstrates the effectiveness of LLM-generated rationales in enhancing alignment with human raters. Semantic similarity and hit-rate analyses indicate that the rationales closely reflect the rubric-criteria, particularly for meaning-focused traits. Additionally, the inclusion of rationales yields improved correlations between predicted and human-rated scores, and factor analysis of the predicted scores reveals a clear distinction between form- and meaning-focused traits. The findings suggest that rubric-based rationales enhance the validity and accuracy of automated essay scoring in EFL contexts.
Abstract
Essay scoring has garnered increasing attention in the field of EFL education due to its efficacy in assessing learners’ cognitive and communicative competencies. However, manually scoring essays often encounters significant bottlenecks, including a lack of scoring expertise, inconsistencies among raters, and substantial time demands. To tackle these challenges, this study proposes a Large Language Model-based Multi-Trait Essay Scoring (LMES) framework, which incorporates rubric-based criteria into the reasoning process of an LLM to generate textual justifications, referred to as rationales. These rationales subsequently guide a BERT language model in predicting trait-specific scores. Utilizing the Feedback Prize essay dataset, LMES demonstrates the effectiveness of LLM-generated rationales in enhancing alignment with human raters. Semantic similarity and hit-rate analyses indicate that the rationales closely reflect the rubric-criteria, particularly for meaning-focused traits. Additionally, the inclusion of rationales yields improved correlations between predicted and human-rated scores, and factor analysis of the predicted scores reveals a clear distinction between form- and meaning-focused traits. The findings suggest that rubric-based rationales enhance the validity and accuracy of automated essay scoring in EFL contexts.