A Study of the Rater Training Effect in English Speaking Assessment
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
This paper reports the findings of an experimental study on the effect of the rater training on L1 and L2 English teacher raters in the criteria application. This study answers to the question of whether the rater training has an effect on their ratings in their test scores based on the constructs presented. The data for the study are derived from two sources: pre- and post-training ratings. The result analysis shows that the training has a considerable effect on the rating patterns, in which fluency and pragmatic resources increase significantly, while demeanor decreases after the training. It is found that the pre-trained teacher raters tend to be more severe and less consistent in their ratings than the post-trained raters. It is evident that the training has a positive effect on the criteria application. However, the English teacher raters, regardless of their L1 nativeness, still have some rating problems even after the training, although consistency has improved a lot for most constructs. All these results support the conclusion that teacher training is successful in helping the raters to give more reliable scores, but it still has limitations and has to be improved.
Abstract
This paper reports the findings of an experimental study on the effect of the rater training on L1 and L2 English teacher raters in the criteria application. This study answers to the question of whether the rater training has an effect on their ratings in their test scores based on the constructs presented. The data for the study are derived from two sources: pre- and post-training ratings. The result analysis shows that the training has a considerable effect on the rating patterns, in which fluency and pragmatic resources increase significantly, while demeanor decreases after the training. It is found that the pre-trained teacher raters tend to be more severe and less consistent in their ratings than the post-trained raters. It is evident that the training has a positive effect on the criteria application. However, the English teacher raters, regardless of their L1 nativeness, still have some rating problems even after the training, although consistency has improved a lot for most constructs. All these results support the conclusion that teacher training is successful in helping the raters to give more reliable scores, but it still has limitations and has to be improved.
참고문헌 (21)
- Bachman, L., "Investigating variability in tasks and rater judgments in a performance test of foreign language speaking", Language Testing, 12(2), pp.238~256 (1995)
- Barnwell, D, "Naive native speakers and judgements of oral proficiency in Spanish", Language Testing, 6(2), pp.152~163 (1989)
- Bonk, W., "A many-facet Rasch analysis of the second language group oral discussion task", Language Testing, 20(1), pp.89~110 (2003)
- Brown, A., "The effect of rater variables in the development of an occupation-specific language performance test", Language Testing, 12(1), pp.1~15 (1995)
- Brown, A., "IELTS Research Reports 2000 (3)", pp.49~84 (2000), IELTS Australia
- An examination of rater orientations and test-taker performance on English-for-Academic-Purposes speaking tasks (No. RR-05-05), "Brown, A.", ETS, ~, pp.Monograph Series (29) (2005)
- Council of Europe, "Common European framework of reference for languages: learning, teaching, assessment" (2001), Cambridge University Press
- Grove, E., "Tasks and criteria in a test of oral communication skills for first-year health science students : Where from", Melbourne Papers in Language Testing, 10(1), pp.37~47 (2001)
- Elder, C., "Language testing and English as an international language", Australian Review of Applied Linguistics, 31(3), pp.31~34 (2008)
- Hinkel, E., "Native and pre-native speakers’ pragmatic interpretations of English texts", TESOL Quarterly, 28(2), pp.353~376 (1994)
- Hudson, T., "The companion to language assessment", pp.561~577 (2013), John Wiley & Sons
- 강석한, "A Study of the Korean English Teachers’ Rating Tendency to the Oral English Proficiency Tests", 영어영문학, 21(4), pp.237~264 (2016), 미래영어영문학회
- 강석한, "A Comparative Study on Criteria and Tasks in Korean English Speaking Assessment by Native and Non-native Raters", 어학연구, 48(2), pp.241~262 (2012), 언어교육원
- Kobayashi, T., "Native and prenative reactions to ESL compositions", TESOL Quarterly, 26(1), pp.81~112 (1992)
- Kunnan, A., "Fairness and validation in language assessment: selected papers from the 19th language testing colloquium" (2000), University of Cambridge Press
- Linacre, J., "Facets Rasch measurement computer program" (2014), Winsteps. com
- Luoma, S., "Assessing speaking" (2004), Cambridge University Press
- Lumley, T., "Rater characteristics and rater bias : Implication for training", Language Testing, 12, pp.54~71 (1995)
- Shohamy, E., "The effect of raters’ background and training on the reliability of direct writing tests", The Modern Language Journal, 76(1), pp.27~33 (1992)
- Weigle, S., "Using FACETS to model rater training effects", Language Testing, 15, pp.263~287 (1998)
- Zhang, Y., "Judgements of oral proficiency by pre-native and native English speaking teacher raters: Competing or complementary constructs?", Language Testing, 28(1), pp.31~50 (2011)