In-Context Learning을 통한 서·논술형 평가에서의 대형언어 모델과 교사 간 채점 및 피드백 정합성 향상 방안
Enhancing Alignment Between Large Language Models and Teacher in Open-Ended Assessment through In-Context Learning
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
This study investigates the effectiveness of in-context learning (ICL) in enhancing theagreement between human teachers and large language models (LLMs) in the context ofopen-ended assessments. Using a dataset of 485 student responses to six open-endedquestions from Korean, Technology, and Social Studies subjects administered in 2024,teacher-generated scores and feedback were collected alongside LLM-generated outputsunder varying ICL conditions. Specifically, we provided GPT-4.1 with 0 to 20 examples inprompts to examine whether increasing example count improves agreement betweenthe model and human raters. Quadratic Weighted Kappa (QWK) was used to assess scorealignment, and BERTScore measured semantic similarity between teacher and modelfeedback. Regression and mixed-effects analyses revealed that increasing the numberof examples generally improved alignment up to a certain threshold. The strongestimprovements occurred with fewer than six examples, beyond which the benefits plateauedor even declined. Additionally, prompt length negatively moderated the effect of examplecount, suggesting that longer prompts may reduce the model’s capacity to focus on relevantinformation. These results provide practical guidance for teachers using LLMs in openendedassessments. Including teacher-generated examples in prompts helps models alignmore closely with human scoring and feedback. However, the optimal number of examplesdepends on the type of question and expected answer length: more examples benefitshorter responses, while fewer examples (five or fewer) are more effective for longer or morecomplex answers.
Abstract
This study investigates the effectiveness of in-context learning (ICL) in enhancing theagreement between human teachers and large language models (LLMs) in the context ofopen-ended assessments. Using a dataset of 485 student responses to six open-endedquestions from Korean, Technology, and Social Studies subjects administered in 2024,teacher-generated scores and feedback were collected alongside LLM-generated outputsunder varying ICL conditions. Specifically, we provided GPT-4.1 with 0 to 20 examples inprompts to examine whether increasing example count improves agreement betweenthe model and human raters. Quadratic Weighted Kappa (QWK) was used to assess scorealignment, and BERTScore measured semantic similarity between teacher and modelfeedback. Regression and mixed-effects analyses revealed that increasing the numberof examples generally improved alignment up to a certain threshold. The strongestimprovements occurred with fewer than six examples, beyond which the benefits plateauedor even declined. Additionally, prompt length negatively moderated the effect of examplecount, suggesting that longer prompts may reduce the model’s capacity to focus on relevantinformation. These results provide practical guidance for teachers using LLMs in openendedassessments. Including teacher-generated examples in prompts helps models alignmore closely with human scoring and feedback. However, the optimal number of examplesdepends on the type of question and expected answer length: more examples benefitshorter responses, while fewer examples (five or fewer) are more effective for longer or morecomplex answers.