An Examination of Raters' Decision-Making Processes on English-speaking Evaluations
2
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
There has been growing interest over the past two decades in investigating raters' rating processes through the analysis of verbal reports. The present study attempts to examine what features skilled native English-speaking raters attend to while they are evaluating nonnative English-speaking speakers' responses. This study employed think-aloud verbal reports made by four raters as they scored 10 nonnative English-speaking test-takers. There were 652 verbal protocols that occurred in total. The rating behaviors are categorized into 25 coding systems. The frequency analysis results from raters' verbal reports have showed that native English-speaking raters often heeded aspects of the performances that are relevant to the assessment criteria. Experienced raters tended to provide more verbal protocols for the evaluation of accuracy judgments compared to the other scoring criteria. While they assessing fluency, raters were most likely to rely more on pauses, hesitations or speed than contents, the number of repetitions or self-corrections. When raters made pronunciation judgments, they raters seemed to consider the degree of Korean accent the most. However, raters sometimes paid attention to features that were somewhat irrelevant to scoring criteria. Based on the findings, theoretical and practical implications are discussed.
Abstract
There has been growing interest over the past two decades in investigating raters' rating processes through the analysis of verbal reports. The present study attempts to examine what features skilled native English-speaking raters attend to while they are evaluating nonnative English-speaking speakers' responses. This study employed think-aloud verbal reports made by four raters as they scored 10 nonnative English-speaking test-takers. There were 652 verbal protocols that occurred in total. The rating behaviors are categorized into 25 coding systems. The frequency analysis results from raters' verbal reports have showed that native English-speaking raters often heeded aspects of the performances that are relevant to the assessment criteria. Experienced raters tended to provide more verbal protocols for the evaluation of accuracy judgments compared to the other scoring criteria. While they assessing fluency, raters were most likely to rely more on pauses, hesitations or speed than contents, the number of repetitions or self-corrections. When raters made pronunciation judgments, they raters seemed to consider the degree of Korean accent the most. However, raters sometimes paid attention to features that were somewhat irrelevant to scoring criteria. Based on the findings, theoretical and practical implications are discussed.
참고문헌 (23)
- Bachman, L. F., "Fundamental considerations in language testing" (1990), Oxford University Press
- Bachman, L. F., "Investigating variability in tasks and rater judgments in a performance test of foreign language speaking", Language Testing, 12(2), pp.239~257 (1995)
- Barnwell, D., "Naive native speakers and judgements of oral proficiency in Spanish", Language Testing, 6(2), pp.152~163 (1989)
- Brown, A., "The effect of rater variables in the development of an occupation specific language performance test", Language Testing, 12(1), pp.277~303 (1995)
- Brown, A., "An examination of the rating process in the revised IELTS Speaking Test", IELTS Research Report, 6, pp.1~30 (2003)
- Cumming, A., "Expertise in evaluating second language compositions", Language Testing, 7(1), pp.31~51 (1990)
- Cumming, A., "Scoring TOEFL essays and TOEFL 2000 prototype writing tasks: An investigation into raters' decision making and development of a preliminary analytic framework (TOEFL Monograph series, MS-22)" (2001), Education Testing Service
- Ericsson, K. A., "Protocol analysis: Verbal reports as data" (1982), MIT Press
- Douglas, D., "Quantity and quality in speaking test performance", Language Testing, 11(1), pp.125~143 (1994)
- Fayer, J. M., "Native and nonnative judgements of intelligibility and irritation", Language Learning, 37(3), pp.313~326 (1987)
- Galloway, V. B., "Perceptions of the communicative efforts of American students of Spanish", The Modern Language Journal, 64(4), pp.428~433 (1980)
- Green, A., "Verbal protocol analysis in language testing research: A handbook" (1998), Cambridge University Press
- Huot, B. A., "The validity of holistic scoring: A comparison of the talk-aloud protocols of expert and novice holistic raters", 박사, pp.1988
- Kim, Y. H., "An investigation into native and non-native teachers' judgments of an English performance : A mixed methods approach", Language Testing, 26(2), pp.187~217 (2009)
- Lumley, T., "Assessment criteria in a large-scale writing test : What do they really mean to the raters?", Language Testing, 19(3), pp.246~276 (2002)
- Lumley, T., "Rater characteristics and rater bias : Implications for training", Language Testing, 12(1), pp.54~71 (1995)
- Lynch, B. K., "Using G-theory and many-facet Rasch measurement in the development in the development of performance assessments of the ESL speaking skills of immigrants", Language Testing, 15(2), pp.158~180 (1998)
- Orr, M., "The FCE speaking test : Using rater reports to help interpret test scores", System, 30(2), pp.143~154 (2002)
- Shin, D. I., "Validation process of an EFL speaking test on G-theory and other analytical techniques", Journal of the Applied Linguistics Association of Korea, 17(1), pp.199~221 (2001)
- Vaughan, C., "Holistic assessment: What goes on in the raters' minds?, In Assessing second language writing in academic contexts", pp.111~125 (1991), Ablex Publishing Corporation
- Weigle, S. C., "Effects of training on raters of ESL compositions", Language Testing, 11(2), pp.197~223 (1994)
- Upshur, J. A., "Systematic effects in the rating of second language speaking ability : Test method and learner discourse", Language Testing, 16(1), pp.82~111 (1999)
- 유경아, "The effect of raters’ language background on English-speaking test ratings across test-takers’ oral proficiency levels", 응용언어학, 26(4), pp.395~419 (2010), 한국응용언어학회