거대언어모델(LLM)을 활용한 한국어 학습자 쓰기 평가 가능성 탐색 연구 - GPT-4o와 GPT-4.5를 활용하여 -
Exploring the Potential of Using Large Language Models (LLMs) for Evaluating Korean Learners’ Writing: A Comparison GPT-4o and GPT-4.5
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
This study explores the possibility of using Large Language Models (LLMs) as tools for evaluating Korean writing by examining the agreement between human raters and GPT models (GPT-4o, GPT-4.5). Intraclass correlation coefficients (ICCs) were computed to assess the level of agreement between scores assigned by human raters and the GPT models on essays written by Korean learners. The ICC for total scores between human raters and GPT-4o was .664, while the ICC with GPT-4.5 was .794, indicating good and excellent agreement, respectively. However, both models showed poor agreement with human raters in “sociolinguistic competence” and “linguistic diversity.” Kruskal-Wallis H tests with post-hoc analyses also revealed significant differences in scores among the three raters in most evaluation categories. Although GPT models demonstrated human-like tendencies in some areas, clear discrepancies were observed in others. These findings suggest that GPT models have considerable potential as supplementary tools in automated writing evaluation, though careful attention to evaluation domains and model behavior is required.
Abstract
This study explores the possibility of using Large Language Models (LLMs) as tools for evaluating Korean writing by examining the agreement between human raters and GPT models (GPT-4o, GPT-4.5). Intraclass correlation coefficients (ICCs) were computed to assess the level of agreement between scores assigned by human raters and the GPT models on essays written by Korean learners. The ICC for total scores between human raters and GPT-4o was .664, while the ICC with GPT-4.5 was .794, indicating good and excellent agreement, respectively. However, both models showed poor agreement with human raters in “sociolinguistic competence” and “linguistic diversity.” Kruskal-Wallis H tests with post-hoc analyses also revealed significant differences in scores among the three raters in most evaluation categories. Although GPT models demonstrated human-like tendencies in some areas, clear discrepancies were observed in others. These findings suggest that GPT models have considerable potential as supplementary tools in automated writing evaluation, though careful attention to evaluation domains and model behavior is required.