Exploring Rater Behaviors During a Writing Assessment Discussion
2
조회
0
다운로드
6
인용
0
참고문헌
키워드
초록
This study explores a discussion held among raters for a writing assessment in a Korean university. It investigates rater behaviors that influence their decision-making processes as revealed in their interaction during discussion. Four raters independently assessed student writing samples using CEFR scales and then held a discussion session to agree on a single score for each sample. Observation and analysis of the rater discussion showed that there were differences in the degree to which individual raters’ initial judgments were reflected in the final decisions and that each rater’s argument style affected the degree. Raters’ personality dynamics, appreciation of student effort, and comprehension of students’ intended meaning were found to be prominent factors that influenced the process of score decisions. These findings have important implications for the use of discussion in performance assessment and for the rating process in general.
Abstract
This study explores a discussion held among raters for a writing assessment in a Korean university. It investigates rater behaviors that influence their decision-making processes as revealed in their interaction during discussion. Four raters independently assessed student writing samples using CEFR scales and then held a discussion session to agree on a single score for each sample. Observation and analysis of the rater discussion showed that there were differences in the degree to which individual raters’ initial judgments were reflected in the final decisions and that each rater’s argument style affected the degree. Raters’ personality dynamics, appreciation of student effort, and comprehension of students’ intended meaning were found to be prominent factors that influenced the process of score decisions. These findings have important implications for the use of discussion in performance assessment and for the rating process in general.
참고문헌 (26)
- Baker, B. A., "Individual differences in rater decision-making style : An exploratory mixed-methods study", Language Assessment Quarterly, 9, pp.225~248 (2012)
- Barkaoui, K., "Variability in ESL essay rating process : The role of rating scale and rater experience", Language Assessment Quarterly, 7, pp.54~74 (2010)
- Broad, B., "Reciprocal authorities in communal writing assessment: Constructing textual value within a “New Politics of Inquiry.”", Assessing Writing, 4(2), pp.133~167 (1997)
- Choi, Y., "FACETS analysis of effects of rater training on secondary school English teachers’ scoring of English writing", Journal of the Applied Linguistics Association of Korea, 18(1), pp.257~292 (2002)
- "Common European framework of reference for languages: Learning, teaching, assessment (CEFR)" (2001)[URL]
- Creswell, J. W., "Research design: Qualitative, quantitative, and mixed methods approaches" (2009), Sage
- Cronbach, L. J., "Generalizability analysis for performance assessments of student achievement or school effectiveness", Educational and Psychological Measurement, 57, pp.373~399 (1997)
- Cumming, A., "Decision-making while rating ESL", EFL writing tasks : A descriptive framework, 2002, pp.86(1), 67~96
- Eckes, T., "Rater types in writing performance assessments : A classification approach to rater variability", Language Testing, 25(2), pp.155~185 (2008)
- Hamp-Lyons., "Rating nonnative writing : The trouble with holistic scoring", TESOL Quarterly, 29(4), pp.759~762 (1995)
- Huot, B., "Toward a new theory of writing assessment", College Composition and Communication, 47(4), pp.549~566 (1996)
- Johnson, J. S., "The influence of rater language background on writing performance assessment", Language Testing, 26(4), pp.485~505 (2009)
- Johnson, R. L., "The relation between score resolution methods and interrater reliability : An empirical study of an analytic scoring rubric", Applied Measurement in Education, 13(2), pp.121~138 (2000)
- Johnson, R. L., "Score resolution : An investigation of the reliability and validity of resolved scores", Applied Measurement in Education, 16(4), pp.299~322 (2003)
- Johnson, R. L., "Resolving score differences in the rating of writing samples: Does discussion improve the accuracy of scores?", Language Assessment Quarterly, 2(2), pp.117~146 (2005)
- Lumley, T., "Assessment criteria in a large-scale writing test: What do they really mean to raters?", Language Testing, 19(3), pp.246~276 (2002)
- Milanovic, M., "Studies in language testing 3: Performance testing, cognition, and assessment", pp.92~114 (1996), University of Cambridge Local Examinations Syndicate
- Moss, P., "Enlarging the dialogue in educational measurement : Voices from interpretative research traditions", Educational Researcher, 25(1), pp.20~28 (1996)
- Moss, P., "An integrative approach to portfolio evaluation for teacher licensure", Journal of Personnel Evaluation in Education, 12(2), pp.139~161 (1998)
- Shi, L., "Native-and nonnative-speaking EFL teachers’ evaluation of Chinese students’ English writing", Language Testing, 18(3), pp.303~325 (2001)
- Shohamy, E., "The effect of raters’ background and training on the reliability of direct writing tests", The Modern Language Journal, 76(1), pp.27~34 (1992)
- Song, B., "Do English and ESL faculty differ in evaluating the essays of native English-speaking and ESL students?", Journal of Second Language Writing, 5, pp.163~182 (1996)
- Turner, C. E., "Listening to the voices of rating scale developers : Identifying salient features for second language performance assessment", The Canadian Modern Language Review, 56(4), pp.555~584 (2000)
- Vaughan, C., "Assessing second language writing in academic contexts", pp.111~126 (1991), Ablex
- Weigle, S. C., "Using FACETS to model rater training effects", Language Testing, 15(2), pp.263~287 (1998)
- Wolfe, E. W., "Cognitive differences in proficient and nonproficient essay scorers", Written Communication, 15, pp.465~492 (1998)