Examination of Rater Training Effect and Rater Eligibility in L2 Performance Assessment
2
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
The purposes of this study were to investigate the effects of rater training in an L2 performance assessment and to examine the eligibility of L2 users of English as raters in L2 performance assessment. Rater training was conducted in order for raters to clearly understand the criteria, the evaluation items, and the evaluation procedure.In this evaluation, the rater training was conducted based on Common European Framework of Reference (CEFR).In the training, the raters watched the videos (North and Hughes, 2003), and discussed the learners' characteristics at each level. The analyses of the evaluations were done before and after the rater training based on Generalizability (G-Theory) and Multifaceted Rasch Analysis (MFRA). In the analyses based on G-Theory, the variance related to the items was reduced to about one sixth after the training, though no difference was found in the rater characteristics before and after the training in the analysis based on MFRA. Comparing the results of Kim (2009) with those of the present study, the raters are equally self-consistent with the raters of native speakers of English in Kim (ibid). Furthermore, it is legitimate to adopt L2 users as the raters, because, in countries where English is a foreign or second language, the non-native users teach and learn English. In this situation, teachers of L2 users are the most appropriate in L2 performance assessment if they are self-consistent in their ratings.
Abstract
The purposes of this study were to investigate the effects of rater training in an L2 performance assessment and to examine the eligibility of L2 users of English as raters in L2 performance assessment. Rater training was conducted in order for raters to clearly understand the criteria, the evaluation items, and the evaluation procedure.In this evaluation, the rater training was conducted based on Common European Framework of Reference (CEFR).In the training, the raters watched the videos (North and Hughes, 2003), and discussed the learners' characteristics at each level. The analyses of the evaluations were done before and after the rater training based on Generalizability (G-Theory) and Multifaceted Rasch Analysis (MFRA). In the analyses based on G-Theory, the variance related to the items was reduced to about one sixth after the training, though no difference was found in the rater characteristics before and after the training in the analysis based on MFRA. Comparing the results of Kim (2009) with those of the present study, the raters are equally self-consistent with the raters of native speakers of English in Kim (ibid). Furthermore, it is legitimate to adopt L2 users as the raters, because, in countries where English is a foreign or second language, the non-native users teach and learn English. In this situation, teachers of L2 users are the most appropriate in L2 performance assessment if they are self-consistent in their ratings.
참고문헌 (33)
- "SST: Standard Speaking Test" (2006)[URL]
- "ACTFL Proficiency Guidelines" (1999)[URL]
- Bachman, L. F., "Investigating variability in tasks and rater judgments in a performance test of foreign language speaking", Language Testing, 12(2), pp.238~257 (1995)
- Bonk, W. J., "A many-facet Rasch analysis of the second language group oral discussion task", Language Testing, 20(1), pp.89~110 (2003)
- Brennan, R. L., "Generalizability theory. ITEMS: The instructional topics in educational measurement series. Module 14", NCME, ~ (1992)
- Canagarajah, A. S., "Non-native educators in English language teaching" (1999), Lawrence Erlbaum Associates
- "Canadian language benchmarks 2000" (2000)[URL]
- Council of Europe, "Common European framework of reference for languages: Learning, teaching, assessment" (2001), CUP
- Crick, J. E., (1984), American College Testing Program
- Embretson, S. E., "Item response theory for psychologists" (2000), Lawrence Erlbaum Associates, Inc
- Ikeda, H., (1994), AsakuraShoten
- Kim, Y., "An investigation into native and non-native teachers’ judgments of oral English performance: A mixed methods approach", Language Testing, 26(2), pp.187~217 (2009)
- Kondo, Y., "A tentative method of reforming your assessment of English abilities into international standards such as Common European Framework of Reference (CEFR) (1): The eligibility of raters and rater training effect in L2 performance assessment", Proceedings of the 15th International Conference of Pan-Pacific Association of Applied Linguistics, pp.436~443 (2010)
- Kondo, Y., "A FACETS analysis of rater bias in measuring Japanese second language writing performance", Language Testing, 19(1), pp.3~31 (2002)
- Kozaki, Y., "Using GENOVA and FACETS to set multiple standards on performance assessment for certification in medical translation from Japanese into English", Language Testing, 21(1), pp.1~27 (2004)
- Lee, J. F., "On hierarchies of reading skills and text types", Modern Language Journal, 72, pp.173~187 (1988)
- Linacre, J. M., (2006), Winsteps. com
- Lumley, T., "Rater characteristics and rater bias: implications for training", Language Testing, 12(1), pp.54~71 (1995)
- Lunz, M. E., "Measuring the impact of judge severity on examination scores", Applied Measurement in Education, 3(4), pp.331~345 (1990)
- McKay, S. L., "Teaching English as an international language" (2002), OUP
- McNamara, T. F., "Measuring second language performance" (1996), Pearson Education Limited
- Myford, C. M., "Introduction to Rasch measurement", pp.460~517 (2004), JAM Press
- Myford, C. M., "Introduction to Rasch measurement", pp.518~574 (2004), JAM Press
- Norcini, J. J., "The credibility and comparability of standards", Applied Measurement in Education, 10(1), pp.39~59 (1997)
- North, B., "CEF illustrative performance samples for relating language examinations to the CEF of languages: Learning, teaching, assessment (CEF) English (Swiss adult learners)", ~ (2003)
- North, B., "Scaling descriptors for language proficiency scales", Language Testing, 14(2), pp.217~263 (1998)
- Salaberry, R., "Revising the revised format of the ACTFL oral proficiency interview", Language Testing, 17, pp.289~310 (2000)
- Takanashi, Y., (2009), Kenkyusha
- Trim, J. L. M., "Some possible lines of development of an overall structure for a European unit credit scheme for foreign language learning by adults", Council of Europe, ~ (1978)
- Weigle, S. C., "Using FACETS to model rater training effects", Language Testing, 15(2), pp.263~287 (1998)
- Widdowson, H. G., "Defining issues in English language teaching" (2003), OUP
- Wright, B. D., "Reasonable mean-square fit values", Raschmeasurement : Transaction of the Rasch MeasurementSIG, 8, pp.370~ (1994)
- Yashiro, K., (2001), Sanshusha