Resolving Discrepant Ratings in Writing Assessments: The Choice of Resolution Method and Its Application
0
조회
0
다운로드
4
인용
0
참고문헌
키워드
초록
When two raters assign discrepant ratings in writing assessments, a method of resolving the differences should be applied to improve the accuracy of scores reported to examinees (operational scores). The present study, in an attempt to locate the most effective resolution method for the current context where not only experienced but also novice raters participate in assessment, examined tertium quid method as compared to averaged original scores of experienced and novice raters. The results of paired t-tests and interrater reliability estimates for seven pairs of raters showed that the ratings of experienced and novice raters were significantly different from each other. To investigate the accuracy of averaged and tertium quid scores, Pearson correlation was conducted by correlating the two resolved scores with criterion score (standard for decision making). Tertium quid scores correlated much higher with the criterion than the averaged scores across seven sets of scores for all six categories (Content, Organization, Style and Quality of Expression, Language Use, Mechanics, and Fluency), thereby providing a positive evidence for tertium quid method as score resolution. The study suggests for future research that it is not sufficient enough to test the efficacy of resolution methods without considering sources of rater variability.
Abstract
When two raters assign discrepant ratings in writing assessments, a method of resolving the differences should be applied to improve the accuracy of scores reported to examinees (operational scores). The present study, in an attempt to locate the most effective resolution method for the current context where not only experienced but also novice raters participate in assessment, examined tertium quid method as compared to averaged original scores of experienced and novice raters. The results of paired t-tests and interrater reliability estimates for seven pairs of raters showed that the ratings of experienced and novice raters were significantly different from each other. To investigate the accuracy of averaged and tertium quid scores, Pearson correlation was conducted by correlating the two resolved scores with criterion score (standard for decision making). Tertium quid scores correlated much higher with the criterion than the averaged scores across seven sets of scores for all six categories (Content, Organization, Style and Quality of Expression, Language Use, Mechanics, and Fluency), thereby providing a positive evidence for tertium quid method as score resolution. The study suggests for future research that it is not sufficient enough to test the efficacy of resolution methods without considering sources of rater variability.
참고문헌 (50)
- Bachman,L.F, "Investigating variability in tasks and rater judgments in a performance test of foreign language speaking", Language Testing, 12(2), pp.238~257 (1995)
- Banerji,M, "Validation of scores", measures from a K-2 developmental assessment in mathematics, 1999, pp.59(4), 694~715
- Barrett,S, "The impact of training on rater variability", International Education Journal, 2(1), pp.49~58 (2001)
- Bonk,W.J, "A many-facet Rasch analysis of the second language group oral discussion task", Language Testing, 20(1), pp.89~110 (2003)
- Brennan,R, "Generalizability of performance assessments.In Technical issues in large-scale performance assessments", pp.19~58 (1996), National Center for Education Statistics
- Brown,A, "The effect of rater variables in the development of an occupation-specific language performance test", Language Testing, 12, pp.1~15 (1995)
- Brown,J.D, "Do English and ESL faculties rate writing samples differently", TESOL Quarterly, 25, pp.587~603 (1991)
- Cherry,R, "Reliability issues in holistic assessment.In Validating holistic scoring for writing assessment:Theoretical and empirical foundations", pp.109~141 (1993), Hampton
- Clauser,B, "Components of rater error in a complex performance assessment", Journal of Educational Measurement, 36, pp.29~45 (1999)
- Coffman,W, "Essay examinations.In Educational measurement(2nd ed.)", pp.271~302 (1971), American Council on Education
- Congdon,P.J, "The stability of rater severity in large-scale assessment programs", Journal of Educational Measurement, 37, pp.163~178 (2000)
- Cumming,A, "Expertise in evaluating second language compositions", Language Testing, 7, pp.31~51 (1990)
- Eckes,T, "Examining rater effects in TestDaF writing and speaking performance assessments:A many-facet Rasch analysis", Language Assessment Quarterly, 2, pp.197~221 (2005)
- Engelhard,G.,Jr., "The measurement of writing ability with a many-faceted Rasch model", Applied Measurement in Education, 5(3), pp.171~191 (1992)
- Engelhard,G.,Jr., "Examining rater errors in the assessment of written composition with a many-faceted Rasch model", Journal of Educational Measurement, 31, pp.93~112 (1994)
- Hamp-Lyons,L, "Writing teachers as assessors of writing.In Exploring the dynamics of second language writing", pp.162~189 (2003), Cambridge University Press
- Herman,J, "A practical guide to alternative assessment" (1992), Association for Supervision and Curriculum Development
- Hieronymous,A, "Handbook for focused holistic scoring" (1987), Riverside
- Hinkel,E, "Native and nonnative speakers’ pragmatic interpretations of English texts", TESOL Quarterly, 28, pp.353~376 (1994)
- Hoyt,W.T, "Magnitude and moderators of bias in observer ratings:A meta-analysis", Psychological Methods, 4, pp.403~424 (1999)
- Johnson,R, "Score resolution:An investigation of the reliability and validity of resolved scores", Applied Measurement in Education, 16, pp.299~322 (2003)
- Johnson,R, "The relationship between score resolution methods and interrater reliability:An empirical study of an analytic scoring rubric", Applied Measurement in Education, 13, pp.121~138 (2000)
- Johnson,R, "Score resolution and the interrater reliability of holistic scores in rating essays", Written Communication, 18, pp.229~249 (2001)
- Johnson, R, "A conceptual framework for score resolution in the rating of performance assessments: The union of validity and reliability", The annual meeting of the American Educational Research Association (2000)
- Jones,R.L, "Second language performance testing:An overview.In Second language performance testing", pp.15~24 (1985), University of Ottawa Press
- Linacre,J.M, (2001), MESA Press
- Lumley,T, "A new approach to standard-setting in language assessment", Melbourne Papers in Language Testing, 3, pp.19~40 (1994)
- Lumley,T, "Rater characteristics and rater bias:Implications for training", Language Testing, 12, pp.54~71 (1995)
- McColly,W, "What does educational research say about the judging of writing ability", Journal of Educational Research, 64, pp.148~156 (1970)
- McNamara,T.F, "Measuring second language performance" (1996), Longman
- Meuffels,B, "De verguisde beoordelaar.Over de deskundigheid van de neerlandicus als opstelbeoordelaar", Tijdschrift voor Taalbeheersing, 11, pp.161~176 (1989)
- Moss,P, "Enlarging the dialogue in educational measurement:Voices from interpretive research traditions", Educational Researcher, 25(1), pp.20~28 (1996)
- Moss, P, "Developing coherence between assessment and reform: An integrative approach to portfolio evaluation for teacher licensure", The annual meeting of the American Educational Research Association (1997)
- Mullis,I, "Scoring direct writing assessments:What are the alternatives", Educational Measurement: Issues and Practices, 3(1), pp.16~18 (1984)
- Myers,M, "A procedure for writing assessment and holistic scoring" (1980), National Council of Teachers of English
- Myford,C, "When raters disagree,then what:Examining a third-rating discrepancy resolution procedure and its utility for identifying unusual patterns of ratings", Journal of Applied Measurement, 3, pp.300~324 (2002)
- Patsula, L, "Resolving rater discrepancies in scoring performance assessments", The annual meeting of the NCME (1998)
- Penny,J, "My life as a reader", Assessing Writing, 8, pp.192~215 (2003)
- Pula,J.J, "A model of background influences on holistic raters.In Validating holistic scoring for writing assessment:Theoretical and empirical foundations", pp.237~265 (1993), Hampton Press
- Schaefer,E, "Rater bias patterns in an EFL writing assessment", Language Testing, 25(4), pp.465~493 (2008)
- Schoonen,R, "Generalizability of writing scores:An application of structural equation modeling", Language Testing, 22, pp.1~30 (2005)
- Schoonen,R, "The assessment of writing ability:Expert readers versus lay readers", Language Testing, 14, pp.157~184 (1997)
- Shohamy,E, "The effect of raters’ background and training on the reliability of direct writing tests", The Modern Language Journal, 76(1), pp.27~33 (1992)
- 송명석, "수행 평가를 적용한 영어 쓰기 능력 향상 방안", 영어어문교육, 8(1), pp.165~197 (2002), 한국영어어문교육학회
- Weigle,S.C, "Using FACETS to model rater training effects", Language Testing, 15, pp.263~287 (1998)
- Weigle,S.C, "Investigating rater", prompt interactions in writing assessment:Quantitative and qualitative approaches, 1999, pp.6, 145~178
- Weigle,S.C, "Assessing writing" (2002), Cambridge University Press
- Weigle,S.C, "Assessing writing" (2003), Cambridge University Press
- Wolcott,W, "An overview of writing assessment:Theory,research,and practice" (1998), National Council of Teachers of English
- Wolfe,E.W, "Cognitive differences in proficient and nonproficient essay scorers", Written Communication, 15, pp.465~492 (1998)