An investigation into pre-service and in-service teachers' judgment of English writing performance
2
조회
0
다운로드
4
인용
0
참고문헌
키워드
초록
The present study aimed to investigate rater characteristics on evaluating writingperformance. Seven raters (three in-service teachers and four pre-service teachers)assigned scores 31 writings of Korean high school students. The writings weregraded using a five-category analytic rating scale (Content, Organization, Vocabulary, Grammar, and Mechanics). The FACETS program revealed rater severity, reliability, and bias patterns among raters. Pre-service teachers, in general, scored leniently, while the most severe and lenient raters were all pre-service teachers. With respect to reliability, the result revealed that only pre-service teachers had rated the essays inconsistently. Also, they did not have high degrees of interrater reliability. In terms of bias, two pre-service teachers had a bias toward Mechanics and Vocabulary, while one in-service teacher toward only Mechanics. These results provide support for the notion that pre-service teachers need more rating exercises before they become in-service teachers.
Abstract
The present study aimed to investigate rater characteristics on evaluating writingperformance. Seven raters (three in-service teachers and four pre-service teachers)assigned scores 31 writings of Korean high school students. The writings weregraded using a five-category analytic rating scale (Content, Organization, Vocabulary, Grammar, and Mechanics). The FACETS program revealed rater severity, reliability, and bias patterns among raters. Pre-service teachers, in general, scored leniently, while the most severe and lenient raters were all pre-service teachers. With respect to reliability, the result revealed that only pre-service teachers had rated the essays inconsistently. Also, they did not have high degrees of interrater reliability. In terms of bias, two pre-service teachers had a bias toward Mechanics and Vocabulary, while one in-service teacher toward only Mechanics. These results provide support for the notion that pre-service teachers need more rating exercises before they become in-service teachers.
참고문헌 (35)
- Bachman, L. F., "Investigating variability in tasks and rater judgments in a performance test of foreign language speaking", Language Testing, 12(2), pp.238~256 (1995)
- Barkaoui, K., "Think-aloud protocols in research on essay rating : An empirical study of their veridicality and reactivity", Language Testing, 28(1), pp.51~75 (2011)
- Bonk, W. J., "A many-facet Rasch analysis of the second language group oral discussion task", Language Testing, 20(1), pp.89~110 (2003)
- Brown, A., "The effect of rater variables in the development of an occupationspecific language performance test", Language Testing, 12(1), pp.1~15 (1995)
- Carrell, P. L., "The effect of writers' personalities and raters' personalities on the holistic evaluation of writing", Assessing Writing, 2(2), pp.153~190 (1995)
- Connor-Linton, J., "Looking behind the curtain: What do L2 composition ratings really mean?", TESOL Quarterly, 29(4), pp.762~765 (1995)
- Eckes, T., "Rater types in writing performance assessments : A classification approach to rater variability", Language Testing, 25(2), pp.155~185 (2008)
- Eckes, T., "Introduction to Many-Facet Rasch Measurement" (2011), Peter Lang
- Elder, C., "Evaluating rater responses to an online training program for L2 writing assessment", Language Testing, 24(1), pp.37~64 (2007)
- Hill, K., "Who should be the judge? The use of non-native speakers as raters on a test of English as an international language", Melbourne Papers in Language Testing, 5(2), pp.29~49 (1996)
- Huot, B., "Validating holistic scoring for writing assessment", pp.206~236 (1993), Hampton
- Iwashita, N., "Can we predict task difficulty in an oral proficiency test? Exploring the potential of an information-processing approach to task design", Language Learning, 51(2), pp.401~436 (2001)
- Jonson, J. S., "The influence of rater language background on writing performance assessment", Language Testing, 26(4), pp.485~505 (2009)
- Kim, Youn-Hee., "An investigation into native and non-native teachers' judgments of oral English performances : A mixed methods approach", Language Testing, 26(2), pp.187~217 (2009)
- Knoch, U., "Retraining writing raters online: How does it compare with face-to-face training?", Assessing writing, 12(1), pp.26~43 (2007)
- Kobayashi, H., "Pragmatic competence and foreign language teaching", pp.161~184 (2003), Universitat Jaume I
- Kondo-Brown, K., "A FACETS analysis of rater bias in measuring Japanese second language writing performance", Language Testing, 19(1), pp.3~31 (2002)
- Lim, G. S., "The development and maintenance of rating quality in performance writing assessment : A longitudinal study of new and experienced raters", Language Testing, 28(4), pp.543~560 (2011)
- Linacre, J. M., "A user's guide to WINSTEPS : Rasch-model computer programs" (2011), Winsteps. com
- Linn, R., "Measurement and assessment in teaching" (2000), Prentice Hall
- Lumley, T., "Assessment criteria in a large scale writing test: What do they really mean to the raters?", Language Testing, 19(3), pp.246~276 (2002)
- Lumley, T., "Assessing second language writing : The rater's perspective" (2005), Peter Lang
- Lynch, B. K., "Using G-theory and Many-facet Rasch Measurement in the development of performance assessments of the ESL speaking skills of immigrants", Language Testing, 15(2), pp.158~180 (1998)
- McNamara, T. F., "Measuring second language performance" (1996), Longman
- McNamara, T. F., "The effect of interlocutor and assessment mode variables in overseas assessments of speaking skills in occupational settings", Language Testing, 14(2), pp.140~156 (1997)
- 박묘영, "Exploring the Raters’ Bias on an EFL Writing Assessment Using Multi-faceted Rasch Measurement", Studies in English Education, 17(2), pp.175~202 (2012), 글로벌영어교육학회
- Shaw, S. D., "Examining writing : Research and practice in assessing second language writing" (2007), Cambridge University Press
- Shi, L., "Native-and nonnative-speaking EFL teachers' evaluation of Chinese students' English writing", Language Testing, 18(3), pp.303~325 (2001)
- 신유선, "A FACETS Analysis of Rater Characteristics and Rater Bias in Measuring L2 Writing Performance", 영어어문교육, 16(1), pp.123~142 (2010), 한국영어어문교육학회
- Sweelder-Brown, C. O., "The influence of training and experience on holistic essay evaluations", English Journal, 75(5), pp.49~55 (1985)
- Weigle, S. C., "Using FACETS to model rater training effects", Language Testing, 15(2), pp.263~267 (1998)
- Weigle, S. C., "Assessing writing" (2002), Cambridge University Press
- Weir, C. J., "Communicative Language Testing" (1990), Prentice Hall
- Wigglesworth, G., "An investigation of planning time and proficiency level on oral test discourse", Language Testing, 14(1), pp.85~106 (1997)
- Wolfe, E., "Cognitive differences in proficient and nonproficient essay scorers", Written Communication, 15(4), pp.465~492 (1998)