Investigating the Effect of Training on Raters’ Bias toward Test Takers in Oral Proficiency Assessment: A FACETS Analysis
1
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
Typically, variability among raters in scoring and their bias is mediated through rater training. However, questions still remain about whether training can affect raters’ severity or leniency. Furthermore, few studies have looked at the differences between trained and untrained raters in oral assessment. Oral test scores of 200 test takers rated by 20 raters and were analyzed before and after a training program using the multifaceted Rasch measurement (MFRM). The results demonstrated the constructive impact of training programs in reducing raters’ biases and increasing their consistency measures. This study indicated that inexperienced raters benefited more from a training program than experienced raters and thus achieved higher measures of consistency afterward. It also demonstrated a higher biased interaction for test takers on the extreme ends of the oral ability continuum. The findings demonstrated that it is almost impossible to completely eradicate rater variability even through rater training. Therefore, rater training should be viewed as a procedure to establish within-rater consistency rather than between-rater consistency. Since this study showed that inexperienced raters can rate even more reliably than experienced ones after training, there is no evidence whereby decision makers can exclude inexperienced raters solely because of their lack of adequate experience. Consequently, decision makers need to use their budgets for establishing rater training programs for inexperienced raters instead.
Abstract
Typically, variability among raters in scoring and their bias is mediated through rater training. However, questions still remain about whether training can affect raters’ severity or leniency. Furthermore, few studies have looked at the differences between trained and untrained raters in oral assessment. Oral test scores of 200 test takers rated by 20 raters and were analyzed before and after a training program using the multifaceted Rasch measurement (MFRM). The results demonstrated the constructive impact of training programs in reducing raters’ biases and increasing their consistency measures. This study indicated that inexperienced raters benefited more from a training program than experienced raters and thus achieved higher measures of consistency afterward. It also demonstrated a higher biased interaction for test takers on the extreme ends of the oral ability continuum. The findings demonstrated that it is almost impossible to completely eradicate rater variability even through rater training. Therefore, rater training should be viewed as a procedure to establish within-rater consistency rather than between-rater consistency. Since this study showed that inexperienced raters can rate even more reliably than experienced ones after training, there is no evidence whereby decision makers can exclude inexperienced raters solely because of their lack of adequate experience. Consequently, decision makers need to use their budgets for establishing rater training programs for inexperienced raters instead.
참고문헌 (39)
- Attali, Y., "A comparison of newly-trained and experienced raters on a standardized writing assessment", Language Testing, 33(1), pp.99~115 (2016)
- Bachman, L. F., "Language testing in practice: Designing and developing useful language tests" (1996), Oxford University Press
- Barkaoui, K., "Think-aloud protocols in research on essay rating: An empirical study on their veridicality and reactivity", Language Testing, 28(1), pp.51~75 (2011)
- Barrett, S., "The impact of training on rater variability", International Education Journal, 2(1), pp.49~58 (2001)
- Bijani, H., "Raters’ perception and expertise in evaluating second language compositions", The Journal of Applied Linguistics, 3(2), pp.69~89 (2010)
- Bijani, H., "The effects of rater training on raters’ severity and bias analysis in second language writing", Iranian Journal of Language Testing, 1(1), pp.1~16 (2011)
- Carey, M. D., "Does a rater familiarity with a candidate’s pronunciation affect the rating in oral proficiency interviews?", Language Testing, 28(2), pp.201~219 (2011)
- Chalhoub-Deville, M., "Deriving oral assessment scales across different tests and rater groups", Language Testing, 12(1), pp.16~33 (1995)
- Cohen, L., "Research methods in education" (2007), Routledge
- Davis, L., "The influence of training and experience on rater performance in scoring spoken language", Language Testing, 33(1), pp.117~135 (2016)
- Eckes, T., "Introduction to many-facet Rasch measurement" (2015), Peter Lang Edition
- Elder, C., "Evaluating rater responses to an online training program for L2 writing assessment", Language Testing, 24(1), pp.37~64 (2007)
- Farhady, H., "Language assessment policy in Iran", Annual Review of Applied Linguistics, 29(1), pp.132~141 (2009)
- Fulcher, G., "Some priority areas for oral language testing", Language Testing Update, 15(1), pp.39~47 (1994)
- Fulcher, G., "Effective rating scale development for speaking tests : Performance decision trees", Language Testing, 28(1), pp.5~29 (2011)
- Gan, Z., "Interaction in group oral assessment : A case study of higher-and lower-scoring students", Language Testing, 27(4), pp.585~602 (2010)
- Huang, H., "Test-taker characteristics and integrated speaking test performance : A path-analytic study", Language Assessment Quarterly, 13(4), pp.283~301 (2016)
- Khabbazbashi, N., "Topic and background knowledge effects on performance in speaking assessment", Language Testing, 34(1), pp.23~48 (2017)
- Kim, H. J., "Investigating raters’ development of rating ability on a second language speaking assessment", doctor, pp.2011
- Kim, H. J., "A qualitative analysis of rater behavior on an L2 speaking assessment", Language Assessment Quarterly, 12(3), pp.239~261 (2015)
- 김연희, "Score Dependability in ESL Oral Performance Assessment: A Generalizability Theory Approach", Journal of Asia TEFL, 14(3), pp.564~572 (2017), 아시아테플
- Knoch, U., "Investigating the effectiveness of individualized feedback to rating behavior : A longitudinal study", Language Testing, 28(2), pp.179~200 (2011)
- Kondo-Brown, K., "A FACETS analysis of rater bias in measuring Japanese second language writing performance", Language Testing, 19(1), pp.3~31 (2002)
- Kuiken, F., "Raters’ decisions, rating procedures and rating scales", Language Testing, 31(3), pp.279~284 (2014)
- Kyle, K., "Construct validity in TOEFL iBT speaking tasks : Insights from natural language processing", Language Testing, 33(3), pp.319~340 (2016)
- Linacre, J. M., "What do infit and outfit, mean-square and standardized mean?", Rasch Measurement Transactions, 16(2), pp.878~ (2002)
- Lumley, T., "Rater characteristics and rater bias : Implications for training", Language Testing, 12(1), pp.54~71 (1995)
- Lunz, M. E., "Judge consistency and severity across grading periods", Evaluation and the Health Professions, 13(4), pp.425~444 (1990)
- McNamara, T. F., "Measuring second language performance" (1996), Longman
- Nakatsuhara, F., "Effect of test-taker characteristics and the number of participants in group oral tests", Language Testing, 28(4), pp.483~508 (2011)
- Saito, H., "EFL classroom peer assessment : Training effects on rating and commenting", Language Testing, 25(4), pp.553~581 (2008)
- Schaefer, E., "Rater bias patterns in an EFL writing assessment", Language Testing, 25(4), pp.465~493 (2008)
- Trace, J., "Measuring the impact of rater negotiation in writing performance assessment", Language Testing, 34(1), pp.3~22 (2017)
- Van Moere, A., "A psycholinguistic approach to oral language assessment", Language Testing, 29(3), pp.325~344 (2012)
- Winke, P., "Raters’ L2 background as a potential source of bias in rating oral performance", Language Testing, 30(2), pp.231~252 (2012)
- Winke, P., "The influence of second language experience and accent familiarity on oral proficiency rating : A qualitative investigation", TESOL Quarterly, 47(4), pp.762~789 (2013)
- Wright, B. D., "Reasonable mean-square fit values", Rasch Measurement Transactions, 8(3), pp.369~386 (1994)
- Xi, X, "Using raters from India to score a large-scale speaking test", Language Learning, 61(4), pp.1222~1255 (2011)
- Zhang, B., "Judgments of oral proficiency by non-native and native English speaking teacher ratings: competing or complementary constructs?", Language Testing, 28(1), pp.31~50 (2011)