국가수준 학업성취도 평가 서답형 문항에 대한 자동채점의 실용성 분석

Abstract

This study aims to evaluate the costs and accuracy of automated scoring for short-answer questions of National Assessment of Educational Achievement (NAEA). To contrast automated and human scoring, both scoring methods were used to grade 14 short-answer questions of the NAEA Social Studies subtest that was taken by 7,442 ninth-grade students in 2014. We analyzed the effects of automated scoring on the costs and accuracy of scoring short-answer questions. We found that more than 60% of the human scoring costs can be saved by only using the automated scoring program. In addition, more than 80% of the human scoring costs can be saved if the current scoring method of NAEA becomes optimized for automated scoring. Correlation coefficients between human and machine scores for 14 questions ranged from 0.97 to 1, exact agreement ranged from 97.76% to 99.9%, and kappa coefficients ranged from 0.94 to 1. Exact and adjacent agreement for 5 questions ranged from 98.15% to 100%, and linear kappa coefficients and quadratic kappa coefficients ranged from 0.96 to 1. Moreover, results showed that correlations and agreement rates between human and machine scores are as high as the ones between human scores. We concluded that automated scoring can reduce the costs of scoring short-answer NAEA questions without sacrificing the accuracy of scoring. It is suggested that the automated scoring system can be used to replace one or two human raters when scoring the short answer NAEA questions, to monitor human scoring, or to train human raters