The Effects of a Rubric on Inexperienced Raters’ Scoring Consistency
1
조회
0
다운로드
2
인용
0
참고문헌
키워드
초록
This study aimed to explore whether a rubric was effective for promoting raters’ scoring consistency when evaluating English learners’ essays. The experienced raters’ scores of Korean university students, with a rubric, were analyzed qualitatively and quantitatively. The scores of inexperienced raters with the rubric were compared with those without the rubric. Each group’s scores were compared using t-tests (also with inter-rater reliability tests) and the inexperienced groups’ scores were compared using ANOVA. With the rubric, the inexperienced raters showed a similar way of evaluating students’ performance to the experienced raters while those without the rubric could not. The former scored the essays consistently both within and across the semesters, but the latter could do so only within the semesters (post-test scores were higher than pre-test scores within each semester). The scores of each test given by the inexperienced raters with the rubric were not statistically different, which was not true for the experienced with the rubric. This implies that rubrics are effective in promoting inexperienced raters’ scoring consistency, especially when raters evaluate essays at the same time rather than when evaluating at different times even with a rubric. The details of the results are given, and the significance is discussed.
Abstract
This study aimed to explore whether a rubric was effective for promoting raters’ scoring consistency when evaluating English learners’ essays. The experienced raters’ scores of Korean university students, with a rubric, were analyzed qualitatively and quantitatively. The scores of inexperienced raters with the rubric were compared with those without the rubric. Each group’s scores were compared using t-tests (also with inter-rater reliability tests) and the inexperienced groups’ scores were compared using ANOVA. With the rubric, the inexperienced raters showed a similar way of evaluating students’ performance to the experienced raters while those without the rubric could not. The former scored the essays consistently both within and across the semesters, but the latter could do so only within the semesters (post-test scores were higher than pre-test scores within each semester). The scores of each test given by the inexperienced raters with the rubric were not statistically different, which was not true for the experienced with the rubric. This implies that rubrics are effective in promoting inexperienced raters’ scoring consistency, especially when raters evaluate essays at the same time rather than when evaluating at different times even with a rubric. The details of the results are given, and the significance is discussed.
참고문헌 (28)
- Bacha, N., "Writing evaluation: What can analytic versus holistic essay scoring tell us?", System, 29(3), pp.371~383 (2001)
- Beyreli, L., "The use of analytic rubric in the assessment of writing performance: Inter-rater concordance study", Educational Sciences : Theory and Practice, 9(1), pp.105~125 (2009)
- Brown, A., "The effect of rater variables in the development of an occupation-specific language performance test", Language Testing, 12(1), pp.1~15 (1995)
- Brown, J. D., "Do English and ESL faculties rate writing samples differently?", TESOL Quarterly, 25(4), pp.587~603 (1991)
- A study on introducing a new system of English speaking lecturers to schools, "Cho, Seokhoon", Ministry of Education, ~ (2008)
- 최연희, "채점자 훈련이 영어 작문 채점에 미치는 효과에 대한 연구: FACETS 분석을 통한 신뢰도 변화 분석", 응용언어학, 18(1), pp.13~292 (2002), 한국응용언어학회
- Hamp-Lyons, L., "Assessing second language writing in academic contexts", pp.5~15 (1991), Ablex Publishing
- Hyland, K., "Teachers’ perceptions of error: The effects of first language and experience", System, 34(4), pp.509~519 (2006)
- Jacobs, H. L., "Testing ESL composition: A practical approach" (1981), Newbury House Publishers
- Knoch, U., "Re-training writing raters online: How does it compare with face-to-face training?", Assessing Writing, 12(1), pp.26~43 (2007)
- Kobayashi, T., "Native and nonnative reactions to ESL compositions", TESOL Quarterly, 26(1), pp.81~112 (1992)
- Kohn, A., "The trouble with rubrics", English Journal, 95(4), pp.12~15 (2006)
- Kondo-Brown, K., "A FACETS analysis of rater bias in measuring Japanese second language writing performance", Language Testing, 19(1), pp.3~31 (2002)
- 이희경, "Native and nonnative rater behavior in grading Korean students' English essays", Asia Pacific Education Review, 10(3), pp.387~397 (2009), 교육연구소[DOI]
- McNamara, T. F., "Measuring second language performance" (1996), Longman
- O’Loughlin, K., "Do English and ESL teachers rate essays differently?", Melbourne Papers in Language Testing, 1(2), pp.19~44 (1992)
- Rezaei, A. R., "Reliability and validity of rubrics for assessment through writing", Assessing Writing, 15(1), pp.18~39 (2010)
- Schaefer, E., "Rater bias patterns in EFL writing assessment", Language Testing, 25(4), pp.465~493 (2008)
- Shi, L., "Native- and nonnative-speaking EFL teachers’ evaluation of Chinese students’ English writing", Language Testing, 18(3), pp.303~325 (2001)
- 신유선, "A FACETS Analysis of Rater Characteristics and Rater Bias in Measuring L2 Writing Performance", 영어어문교육, 16(1), pp.123~142 (2010), 한국영어어문교육학회
- Silvestri, L., "Using rubrics to increase the reliability of assessment in health classes", International Electronic Journal of Health Education, 9, pp.25~30 (2006)
- Song, B., "Do English and ESL faculty differ in evaluating the essays of native English-speaking and ESL students?", Journal of Second Language Writing, 5(2), pp.163~182 (1996)
- Spandel, V., "In defense of rubrics", English Journal, 96(1), pp.19~22 (2006)
- Stevens, D. D., "Introduction to rubrics: An assessment tool to save grading time, convey effective feedback and promote student learning" (2005), Stylus
- Weigle, S. C., "Using FACETS to model rater training effects", Language Testing, 15(2), pp.263~287 (1998)
- Weigle, S. C., "Assessing writing" (2002), Cambridge University Press
- Wigglesworth, G., "Exploring bias analysis as a tool for improving rater consistency in assessing oral interaction", Language Testing, 10(3), pp.305~335 (1993)
- Wolf, K., "The role of rubrics in advancing and assessing student learning", The Journal of Effective Teaching, 7(1), pp.3~14 (2007)