Estimating the Inter-rater Reliability of the English Speaking Proficiency Test under a Generalizability Theory Model

Abstract

Since the evaluation of the language performance tests is done by human raters, there is a lot of concern on scoring consistency, and whether the scores given by raters are reliable. The purpose of this study was to estimate the inter-rater reliability of the English Speaking Proficiency Test(ESPT) from the perspective of the generalizability theory. More specifically, this study attempted to identify major sources of measurement error and the magnitude of each through the estimated variance components. The three English-speaking native teachers participated in this study as raters; two experienced raters and one novice rater. They scored 116 Korean adults' English speaking responses. Their ratings were analysed through the GENOVA program. The results of the GENOVA analyses illustrated that the greatest portion of the total variance was from persons variance and a moderate effect of the tasks on the test scores due to the task difficulty. However, the variance for rater facet was extremely small. One of the interesting findings was that rater’s rating pair influenced the variance. There was a moderate interaction effect between persons and the raters when the raters consisted of one experienced and the other novice as a pair. It indicated that the rater pair containing experienced raters tended to give scores consistently more than the rater pair composing of one experienced and one novice raters.