Does Raters’ Rating Experience Influence English-speaking Test Ratings?
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
In second language performance tests, rater effect has been a major concern in that it is a potential source of measurement error. The purpose of the study is to examine if raters' rating experience affects their English-speaking ratings. Two experienced and two less experienced native English-speaking raters scored 116 Korean college test-takers' English-speaking test responses. The data were analyzed with the GENOVA program and independent samples t-tests. The results revealed that both experienced and less experienced raters yielded almost identical dependability indices, showing that they scored with consistency the English-speaking samples within their group. The findings of this study also indicate that there is no significant rater effect on combined scores between experienced and less experienced raters. However, there is a significant rater experience effect on some analytic rating components such as pronunciation. Based on the findings, theoretical and practical implications are discussed.
Abstract
In second language performance tests, rater effect has been a major concern in that it is a potential source of measurement error. The purpose of the study is to examine if raters' rating experience affects their English-speaking ratings. Two experienced and two less experienced native English-speaking raters scored 116 Korean college test-takers' English-speaking test responses. The data were analyzed with the GENOVA program and independent samples t-tests. The results revealed that both experienced and less experienced raters yielded almost identical dependability indices, showing that they scored with consistency the English-speaking samples within their group. The findings of this study also indicate that there is no significant rater effect on combined scores between experienced and less experienced raters. However, there is a significant rater experience effect on some analytic rating components such as pronunciation. Based on the findings, theoretical and practical implications are discussed.
참고문헌 (35)
- Alderson, J. C., "Language Test Construction and Evaluation" (1995), Cambridge University Press
- Bachman, L. F, "Fundamental Considerations in Language Testing" (1990), Oxford University Press
- Bachman, L. F., "Investigating Variability in Tasks and Rater Judgments in a Performance Test of Foreign Language Speaking", Language Testing, 12, pp.239~257 (1995)
- Barnwell, D, "Naive Native Speakers and Judgements of Oral Proficiency in Spanish", Language Testing, 6(2), pp.152~163 (1989)
- Brennan, R. L, "Generalizability Theory" (2001), Springer
- Brown, A, "The Effect of Rater Variables in the Development of an Occupation Specific Language Performance Test", Language Testing, 12(1), pp.277~303 (1995)
- Crick, J. E, "GEVOVA: A General Purpose Analysis of Variance System. Version 3.1" (2001), American College Testing Program
- Chalhoub-Deville, M, "Rater Judgment and English Language Speaking Proficiency", World Englishes, 24(3), pp.383~391 (2005)
- Choi, Y. H, "GENOVA and FACETS Analysis of English Writing Test: Tasks, Raters and Scales", English Teaching, 56, pp.125~142 (2001)
- Cumming, A, "Expertise in Evaluating Second Language Compositions", Language Testing, 7(1), pp.31~51 (1990)
- Cumming, A., "Scoring TOEFL Essays and TOEFL 2000 Prototype Writing Tasks: An Investigation into Raters' Decision Making and Development of a Preliminary Analytic Framework" (2001), Education Testing Service
- Eckes, T, "Examining Rater Effects in TestDaF Writing and Speaking Performance Assessment: A Many-facet Rasch Analysis", Language Assessment Quarterly, 2(3), pp.197~221 (2005)
- Fayer, J. M, "Native and Nonnative Judgements of Intelligibility and Irritation", Language Learning, 37(3), pp.313~326 (1987)
- Hamp-Lyons, L, "Rating Nonnative Writing: The Trouble with Holistic Scoring", TESOL Quarterly, 29(4), pp.759~762 (1995)
- Johnson, J. S, "The Influence of Rater Language Background on Writing Performance Assessment", Language Testing, 26(4), pp.485~505 (2009)
- 김현정, "An investigation of novice, developing, and experienced raters’ rating patterns on a second language speaking assessment.", 응용언어학, 26(4), pp.151~182 (2010), 한국응용언어학회
- 김종남, "실용영어 평가도구의 비교와 분석", 영어영문학연구, 28(2), pp.177~195 (2002), 대한영어영문학회
- Kim, Y. H, "An Investigation into Native and Non-native Teachers' Judgments of an English Performance: A mixed Methods Approach", Language Testing, 26(2), pp.187~217 (2009)
- 이영식, "다변량 일반화가능도 이론에 의한 말하기 시험의 타당도와 신뢰도에 관한 연구", Foreign Languages Education, 11(2), pp.249~265 (2004), 한국외국어교육학회
- Lee, Y. W, "Dependability of Scores for a New ESL Speaking Assessment Consisting of Integrated and Independent Tasks", Language Testing, 23(2), pp.131~166 (2006)
- Lumley, T, "Assessment Criteria in a Large-scale Writing Test: What do They Really Mean to the Raters?", Language Testing, 19(3), pp.246~276 (2002)
- Lumley, T, "Rater Characteristics and Rater Bias: Implications for Training", Language Testing, 12(1), pp.54~71 (1995)
- Lynch, B. K, "Using G theory and Many-facet Rasch Measurement in the Development of Performance Assessments of the ESL Speaking Skills of Immigrants", Language Testing, 15(2), pp.158~180 (1998)
- McNamara, T. F, "Measuring Second Language Performance" (1996), Longman
- Orr, M, "The FCE Speaking Test: Using Rater Reports to Help Interpret Test Scores", System, 30(2), pp.143~154 (2002)
- Shavelson, R. J., "Generalizability Theory: A primer. Newbury Park" (1991), Sage Publications
- Shi, L, "Native- and Nonnative-speaking EFL Teachers’ Evaluation of Chinese Students’ English Writing", Language Testing, 18(3), pp.303~325 (2001)
- Shin. D. I, "Validation Process of an EFL Speaking Test on G-theory and Other Analytical Techniques", Journal of the Applied Linguistics Association of Korea, 17(1), pp.199~221 (2001)
- 신동일, "Korean Examinees Perceptions of Testing Spoken English: A Move Toward Proficiency", 영어영문학연구, 36(2), pp.263~288 (2010), 대한영어영문학회
- 신유선, "Investigating Variability in Tasks, Analytic Scoring Criteria, and Rater Groups on L2 Writing in English", Foreign Languages Education, 17(2), pp.27~57 (2010), 한국외국어교육학회
- Shohamy, E., "The Effect of Raters' Background and Training on the Reliability of Direct Writing Tests", The Modern Language Journal, 76(1), pp.27~33 (1992)
- Upshur, J. A, "Systematic Effects in the Rating of Second Language Speaking Ability: Test Method and Learner Discourse", Language Testing, 16(1), pp.82~111 (1999)
- Vaughan, C, "Holistic Assessment: What Goes on in the Raters' Minds?, In Assessing Second Language Writing in Academic Contexts", pp.111~125 (1991), Ablex Publishing Corporation
- Weigle, S. C, "Effects of Training on Raters of ESL Compositions", Language Testing, 11, pp.197~223 (1994)
- 유경아, "The effect of raters’ language background on English-speaking test ratings across test-takers’ oral proficiency levels", 응용언어학, 26(4), pp.395~419 (2010), 한국응용언어학회