Rater Training and Recruitment in ESL Assessment: Current Issues and Practices
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Both direct and semi-direct ESL tests which include speaking and/or writing sections necessitate the use of raters. This paper aims to give a summary of some of the issues and practices in the training of raters for such tests, as well as recommendations for the implementation of such training. First, some examples of how raters can be trained and monitored are given; important issues and developments in the field are then looked at. A model for training is then presented together with factors that may influence recruitment and program design, before turning to current practices employed by some of the larger testing organizations. The area of recruitment in particular is investigated in some detail. Despite the difficulty of finding out the exact details of training policies and training sessions, it is apparent that differences in approach between organizations and tests exist. These differences can be attributed to the overall design and aims of the test in question, as well as the practical implications of administering and conducting rater training sessions.
Abstract
Both direct and semi-direct ESL tests which include speaking and/or writing sections necessitate the use of raters. This paper aims to give a summary of some of the issues and practices in the training of raters for such tests, as well as recommendations for the implementation of such training. First, some examples of how raters can be trained and monitored are given; important issues and developments in the field are then looked at. A model for training is then presented together with factors that may influence recruitment and program design, before turning to current practices employed by some of the larger testing organizations. The area of recruitment in particular is investigated in some detail. Despite the difficulty of finding out the exact details of training policies and training sessions, it is apparent that differences in approach between organizations and tests exist. These differences can be attributed to the overall design and aims of the test in question, as well as the practical implications of administering and conducting rater training sessions.
참고문헌 (37)
- "ACTFL OPIc Rater Job Opening"[URL]
- Alderson,J.C., "Language Test Construction and Evaluation" (1995), Cambridge University Press.
- Bachman,L.F., "Language Testing in Practice" (1996), Oxford University Press
- Bachman, L.F., "Investigating variability in tasks and rater judgment in a performance test of foreign speaking", Language Testing, (12), pp.238~57 (1995)
- Bachman,L.F., "Statistical Analyses for Language Assessment" (2004), Cambridge University Press
- Barritt, L., "Researching practice: evaluating assessment essays", College Composition and Communication, (37), pp.315~327 (1986)
- British Council, "A Guide for Applicants to IELTS Examiner Recruitmentfrom the World Wide Web in http:", guide_for_applicants.pdf, 2007[DOI]
- Brown,A, "The effect of rater variables in the development of an occupation-specific language performance test", Language Testing, 12(1), pp.1~5 (1995)
- Revising the IELTS Speaking Test in retraining examiners worldwide, "Cambridge ESOL", Research Notes, 9~11 (2001)
- Cambridge ESOL, "to accompany examiner application forms from the World Wide Web http:", assets, pp.pdf, what-we-do[DOI]
- Charney,D., "The validity of using holistic scoring to evaluate writing in a critical overview", Research in the Teaching of English, (18), pp.65~81 (1984)
- Constable,E., "Inter-judge reliability in is complete agreement among judges the ideal?" (1984), New Orleans LA
- Elder,C., "Evaluating rater responses to an online training program for L2 writing assessment", Language Testing, (24), pp.37~64 (2007)
- ETS, "Frequently Asked Questions about TOEFL iBT Speaking and Writing Online Scoring http:", portal, pp.site, ets[DOI]
- ETS, "Information for Prospective Test of English for International Communication (TOEIC) Raters http:", portal, pp.site, ets[DOI]
- Hamilton S., "Teachers’ perceptions of on-line rater training and monitoring", System, (29), pp.505~520 (2001)
- Hamps-Lyon J., "Worrying about rating", Assessing Writing, (12), pp.1~9 (2007)
- Huot, B, , Reliability, validity, and holistic scoring in What we know, what we need to know, pp.(41), 201~213
- Hughes,A., "Testing for Language Teachers" (2003), Cambridge University Press.
- Test performance 2006. Retreived 31 December, 2007, from the World Wide Web : http:, , www.ielts.org, analysisoftestdata, pp.article380.asp, Test performance 2006 http:
- Knoch U., "Re-training writing raters online in How does it compare with face-to-face training?", Assessing Writing, (12), pp.26~43 (2007)
- Lumley,T., "Rater characteristics and rater bias in Implications for training", Language Testing, (12), pp.54~71 (1995)
- Lumley,T., "Assessment criteria in a large-scale writing test in what do they really mean to the raters?", Language Testing, (19), pp.246~263 (2002)
- Luoma,S., "Assessing Speaking" (2004), Cambridge University Press
- Lunz,M.E., "Measuring the impact of judge severity on examination scores", Measurement in Education, (3), pp.331~345 (1990)
- McNamara,T., "Measuring second language performance" (1996), Pearson Education
- Myford,C.M., "Detecting and measuring rater effects using many-facet Rasch measurement I", Journal of Applied Measurement, 4(4), pp.386~422 (2003)
- Myford, C. M., "Detecting and measuring rater effects using many facet Rasch measurement in Part II. Journal of Applied Measurement", Measurement, 5(2), pp.189~227 (2004)
- "Examiner recruitment"[URL]
- Orr,M., "The FCE Speaking test:using rater reports to help interpret test scores", System, 3(2), pp.143~154 (2002)
- "Online Marking FAQs"[URL]
- Shi,L., "Native-and non-native-speaking EFL teachers’ evaluation of Chinese students’ English writing", Testing, (18), pp.303~323 (2001)
- Shohamy,E., "The Effect of Raters' Background and Training on the Reliability of Direct Writing Tests", Modern Language Journal, 76(1), pp.27~33 (1992)
- Weigle,S.C., "Effects of training on raters of ESL compositions", Language Testing, (11), pp.197~223 (1994)
- Weigle,S.C., "Using facets to model rater training effects", Language Testing, (15), pp.263~287 (1998)
- Weigle,S.C., "Assessing Writing" (2004), Cambridge University Press
- White,E., "Teaching and Assessing Writing" (1985), Jossey-Bass