Effects of Rating Criteria Order on Novice Korean Raters’ Attentional Behaviors in L2 Writing Assessment: A Think-Aloud Protocol Study
3
조회
0
다운로드
0
인용
0
참고문헌
초록
Previous studies on rater variability have shown that the amount of attention directed towards each rating criterion was associated with rating criteria order in analytic rating scales. Thus, this study attempts to explore whether rating criteria order affects the amount of attention directed to rating criteria using think-aloud protocols. To achieve a research goal, 11 novice Korean raters rated two essays in two different rating criteria orders. In the standard-order rating rubric, the rating criteria of textual aspects of the essays, including Content and Organization, were first evaluated. Then, the rating criteria of grammatical aspects of the essays, including Vocabulary and Language use, were evaluated. This rating order was precisely reversed in the reverse-order rating rubric. The overall results of this study show that heightened attention from the raters was directed toward either the first- or last-presented rating criterion depending on the quality of each essay. Specifically, a differential amount of attention was directed towards Content and Language Use when rating a poor-quality essay. A significant implication of the study is that rating criteria order should be considered in developing analytic rating scales, and in training newly recruited raters.
Abstract
Previous studies on rater variability have shown that the amount of attention directed towards each rating criterion was associated with rating criteria order in analytic rating scales. Thus, this study attempts to explore whether rating criteria order affects the amount of attention directed to rating criteria using think-aloud protocols. To achieve a research goal, 11 novice Korean raters rated two essays in two different rating criteria orders. In the standard-order rating rubric, the rating criteria of textual aspects of the essays, including Content and Organization, were first evaluated. Then, the rating criteria of grammatical aspects of the essays, including Vocabulary and Language use, were evaluated. This rating order was precisely reversed in the reverse-order rating rubric. The overall results of this study show that heightened attention from the raters was directed toward either the first- or last-presented rating criterion depending on the quality of each essay. Specifically, a differential amount of attention was directed towards Content and Language Use when rating a poor-quality essay. A significant implication of the study is that rating criteria order should be considered in developing analytic rating scales, and in training newly recruited raters.
참고문헌 (24)
- Bachman, L. F., "Fundamental considerations in language testing" (1990), Oxford University Press
- Bachman, L. F., "Language testing in practice : Designing and developing useful language tests" (1996), Oxford University Press
- Ballard, L., "The eff ects of primacy on rater cognition: An eye-tracking study", Doctor, pp.2017
- Barkaoui, K., "Think-aloud protocols in research on essay rating : An empirical study of their veridicality and reactivity", Language Testing, 28(1), pp.51~ (2011)
- Bowles, M. A., "The think-aloud controversy in second language research" (2010), Routledge
- Cumming, A., "Expertise in evaluating second language compositions", Language Testing, 7(1), pp.31~51 (1990)
- Cumming, A., "Decision making while rating ESL", EFL writing tasks : A descriptive framework, 2002, pp.86(1), 67~96
- DeRemer, M. L., "Writing assessment : Raters’ elaboration of the rating task", Assessing Writing, 5(1), pp.7~29 (1998)
- Eckes, T., "Rater types in writing performance assessments : A classifi cation approach to rater variability", Language Testing, 25(2), pp.155~185 (2008)
- Eckes, T., "Operational rater types in writing assessment : Linking rater cognition to rater behavior", Language Assessment Quarterly, 9(3), pp.270~292 (2012)
- Ericsson, K. A., "Protocol analysis : Verbal reports as data" (1993), MIT Press
- Gass, S. M., "Stimulated recall methodology in second language research" (2000), Erlbaum
- Green, A., "Verbal protocol analysis in language testing research : A handbook(Vol. 5)" (1998), Cambridge University Press
- Ishikawa, S., "Learner corpus studies in Asia and the world", pp.91~118 (2013), Kobe University
- Jacobs, H., "Testing ESL composition : A practical approach" (1981), Newbury House
- 이경랑, "The Effects of a Rubric on Inexperienced Raters’ Scoring Consistency", 현대영어교육, 17(2), pp.75~90 (2016), 현대영어교육학회
- Ling, G., "A study on the impact of fatigue on human raters when scoring speaking responses", Language Testing, 31(4), pp.479~ (2014)
- Lumley, T., "Assessment criteria in a large-scale writing test : What do they really mean to the raters?", Language Testing, 19(3), pp.246~276 (2002)
- Lumley, T., "Assessing second language writing : The rater’s perspective" (2005), Peter Lang
- Mackey, A., "Second language research : Methodology and design" (2015), Routledge
- Weigle, S. C., "Eff ects of training on raters of ESL compositions", Language Testing, 11(2), pp.197~223 (1994)
- Weigle, S. C., "Investigating rater", prompt interactions in writing assessment : Quantitative and qualitative approaches, 1999, pp.6(2), 145~178
- Winke, P., "ESL essay raters’ cognitive processes in applying the Jacobs et al. rubric : An eye-movement study", Assessing Writing, 25, pp.38~54 (2015)
- Wolfe, E. W., "Cognitive differences in proficient and nonproficient essay scorers", Written Communication, 15(4), pp.465~492 (1998)