Rating Performance of EFL Teachers in Writing Assessment: Comparison of Experienced and Novice Raters
2
조회
0
다운로드
9
인용
0
참고문헌
키워드
초록
This paper explores rating performance of high school EFL teachers in national curriculum-specific writing assessment by comparing rater reliability and ratings of experienced and novice raters. The experienced raters completed the government-authorized writing test rater certification program and had two- or three-year rating experience, whereas the novice raters were new teachers who lacked rater training and rating experience. The two groups of teacher-raters rated 410 samples written by high school students in four assessing aspects: task completion, content, organization, and language use. Results reveal the discernible influence of rater background on ratings. The experienced teacher-raters’ rating experience and understanding of the target test task and the rating rubric acquired through the rater certification process resulted in better rater reliability and significantly lower ratings than the novice teacher-raters. Variations among assessment aspects were also noted: for example, lower reliability in content and organization and lower ratings of the experienced teacher-raters in organization and language use. The findings from the study imply that test-takers may receive different ratings depending on who rates their writing, and thus, securing credible experienced raters is essential in a high-stake writing test.
Abstract
This paper explores rating performance of high school EFL teachers in national curriculum-specific writing assessment by comparing rater reliability and ratings of experienced and novice raters. The experienced raters completed the government-authorized writing test rater certification program and had two- or three-year rating experience, whereas the novice raters were new teachers who lacked rater training and rating experience. The two groups of teacher-raters rated 410 samples written by high school students in four assessing aspects: task completion, content, organization, and language use. Results reveal the discernible influence of rater background on ratings. The experienced teacher-raters’ rating experience and understanding of the target test task and the rating rubric acquired through the rater certification process resulted in better rater reliability and significantly lower ratings than the novice teacher-raters. Variations among assessment aspects were also noted: for example, lower reliability in content and organization and lower ratings of the experienced teacher-raters in organization and language use. The findings from the study imply that test-takers may receive different ratings depending on who rates their writing, and thus, securing credible experienced raters is essential in a high-stake writing test.
참고문헌 (37)
- Barkaoui, K., "Do ESL essay raters’ evaluation criteria change with experience?: A mixed-methods, cross-sectional study", TESOL Quarterly, 44, pp.31~57 (2010)
- Barkaoui, K., "Variability in ESL essay rating processes : The role of the rating scale and rater experience", Language Assessment Quarterly, 7, pp.54~74 (2010)
- Brown, J. D., "Do English and ESL faculties rate writing samples differently?", TESOL Quarterly, 25, pp.587~603 (1991)
- Choi, Y. H., "Chajeom-ja hunryeon-i yeongeo jakmun chaejeom-ey michi-neun hyogwa-ey daeha-n yeongu: FACETS bunseok-eul thongha-n sinroey-do byeonhwa bunseok (FACETS analysis of effects of rater training on secondary school English teachers’ scoring of English writing)", Journal of the Applied Linguistics Association of Korea, 18(1), pp.257~292 (2002)
- Cumming, A., "Expertise in evaluating second language compositions", Language Testing, 7, pp.31~51 (1990)
- Cumming, A., "Decision making while rating ESL", EFL writing tasks : A descriptive framework, 2002, pp.86, 67~96
- Eckes, T., "Rater types in writing performance assessments : A classification approach to rater variability", Language Testing, 25, pp.155~185 (2008)
- He, L., "ESL students’ perceptions and experiences of standardized English writing tests", Assessing Writing, 13, pp.130~149 (2008)
- Hughes, A., "Competing criteria for error gravity", ELT Journal, 26, pp.175~182 (1982)
- James, "Judgments of error gravities", ELT Journal, 31, pp.116~124 (1977)
- 장소영, "The Development of a Standardized Rater Training Program on Essay Scoring: A Case Study", 영어교육, 66(3), pp.201~231 (2011), 한국영어교육학회
- Korea Institute of Curriculum and Evaluation, "Hakgyo-danwui yeongeo mal- ha-gi sseu-gi phyeongga munhang chuljey mit chaejeom manual: Godeung-hakgyo (High school English speaking and writing test construction and scoring manual)" (2010), KICE
- Gukga-yeongeo-neungryeok-siheom i-reo-h-gey junbi-ha-s-eyyo: 3-geup haksaeng-yong (NEAT preparation guide book for students: Level 3), "KICE", ~, pp.Research Report No. ORM 2012-54-2 (2012)
- 김보람, "Resolving Discrepant Ratings in Writing Assessments: The Choice of Resolution Method and Its Application", 영어교육, 66(2), pp.211~231 (2011), 한국영어교육학회
- Kobayashi, T., "Native and nonnative reactions to ESL compositions", TESOL Quarterly, 26, pp.81~112 (1992)
- Kobayashi, H., "Factors affecting composition evaluation in an EFL context : cultural rhetorical pattern and readers’’ background", Language Learning, 46, pp.397~437 (1996)
- Kondo-Brown, K., "A FACETS analysis of rater bias in measuring Japanese second language writing performance", Languages Testing, 19, pp.3~31 (2002)
- Lamprianou, I., "High stakes tests with self-selected essay questions : Addressing issues of fairness", International Journal of Testing, 8, pp.55~89 (2008)
- Landis, J. R., "The measurement of observer agreement for categorical data", Biometrics, 33, pp.159~174 (1977)
- 이희경, "Native and nonnative rater behavior in grading Korean students' English essays", Asia Pacific Education Review, 10(3), pp.387~397 (2009), 교육연구소
- 이성용, "Rating of Korean Students’ L2 Writing: Similarities and Differences Between Native and Non-Native Raters", 교과교육학연구, 16(3), pp.629~655 (2012), 교과교육연구소
- Lim, G. S., "The development and maintenance of rating quality in performance writing assessment : A longitudinal study of new and experienced raters", Language Testing, 28, pp.543~560 (2011)
- Lumley, T., "Assessment criteria in a large-scale writing test: What do they really mean to the raters?", Language Testing, 19, pp.246~276 (2002)
- Lumley, T., "second language writing: The rater’s perspective" (2005), Peter Lang
- Lumley, T., "Rater characteristics and rater bias : Implications for training", Language Testing, 12, pp.4~71 (1995)
- Rezaee, A. A., "Essay raters’ personality types and rater reliability", International Journal of Language Studies, 5(4), pp.109~122 (2011)
- Royal-Dawson, L., "Is teaching experience necessary for reliable scoring of extended English questions?", Educational Measurement: Issues and Practice, 28(2), pp.2~8 (2009)
- Ryu, H., "Evaluation of Korean EFL students’ English writing: A comparison of native- and nonnative-speaking teachers’ evaluation", 석사, pp.2011
- Schoonen, R., "The assessment of writing ability : Expert readers versus lay readers", Language Testing, 14, pp.157~184 (1997)
- Shi, L., "Native-and nonnative-speaking EFL teachers’ evaluation of Chinese students’ English writing", Language Testing, 18, pp.303~325 (2001)
- Shohamy, E., "The effect of raters’ background and training on the reliability of direct writing tests", The Modern Language Journal, 76, pp.27~33 (1992)
- Song, C. B., "Do English and ESL faculty differ in evaluating the essays of native English-speaking and ESL students?", Journal of Second Language Writing, 5, pp.163~182 (1996)
- Sweedler-Brown, C. O., "The influence of training and experience on holistic essay evaluation", English Journal, 74, pp.49~55 (1985)
- Valdez Pierce, L., "Performance and portfolio assessment for language minority students" (1992), National Clearing House for Bilingual Education
- Weigle, S. C., "Effects of training on raters of ESL compositions", Language Testing, 11, pp.197~223 (1994)
- Weigle, S. C., "Using FACETS to model rater training effects", Language Testing, 15, pp.263~287 (1998)
- Wiseman, C. S., "Rater effects : Ego engagement in rater decision-making", Assessing Writing, 17, pp.150~173 (2012)