Inter-rater Reliability of English Essay Test and Korean-English Translation Test Across Multiple Topics and Rating Criteria
1
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
The presence and significance of writing performance assessment is growing in standardized English tests in Korea owing to its merits in direct measurement of learners' real-life writing competence. However, English essay writing, the most extensively utilized writing task in language tests in Korea, has often been associated with issues related to its generally low inter-rater reliability. The present study thus aimed to compare the level of inter-scorer agreement between the English essay test and the Korean-English translation test across different topics and rating criteria, with a view to exploring the latter's potentials as a reliable means for direct assessment of learners' writing competence. For this study, 95 Korean university students took both forms of the test under the same topic, and four raters scored their responses solely based on an analytic scoring scale. Analyses of the ratings unveiled higher inter-rater reliability measures for the translation test in both topics and most of the rating criteria. The implications of these findings are discussed together with suggestions for further probing into ways to judge coherence in translated writings.
Abstract
The presence and significance of writing performance assessment is growing in standardized English tests in Korea owing to its merits in direct measurement of learners' real-life writing competence. However, English essay writing, the most extensively utilized writing task in language tests in Korea, has often been associated with issues related to its generally low inter-rater reliability. The present study thus aimed to compare the level of inter-scorer agreement between the English essay test and the Korean-English translation test across different topics and rating criteria, with a view to exploring the latter's potentials as a reliable means for direct assessment of learners' writing competence. For this study, 95 Korean university students took both forms of the test under the same topic, and four raters scored their responses solely based on an analytic scoring scale. Analyses of the ratings unveiled higher inter-rater reliability measures for the translation test in both topics and most of the rating criteria. The implications of these findings are discussed together with suggestions for further probing into ways to judge coherence in translated writings.
참고문헌 (27)
- Bachman, L. F., "Fundamental considerations in language testing" (1990), Oxford University Press
- Bachman, L. F., "Language testing in practice" (1996), Oxford University Press
- Brown, J. D., "A categorical instrument for scoring second language writing skills", Language Learning, 34(4), pp.21~38 (1984)
- Carrell, P. L., "The effect of writers’ personalities and raters’ personalities on the holistic evaluation of writing", Assessing Writing, 2(2), pp.153~190 (1995)
- Choi, I. C., "Practical English grammar encyclopedia" (2010), SaramIn Publishing Co
- Choi, I. C., "Theoretical foundations & practical applications of language assessment", pp.2014
- Choi, I. C., "Comparative analyses of computer-based ratings and human ratings based on performance testing of writing", Multimedia-Assisted Language Learning, 4(1), pp.165~184 (2001)
- Cumming, A., "Assessing integrated writing tasks for academic purposes : Promises and perils", Language Assessment Quarterly, 10, pp.1~8 (2013)
- Eckes, T., "Examining rater effects in TestDaF writing and speaking performance assessments : A Many-facet Rasch analysis", Language Assessment Quarterly, 2, pp.197~221 (2005)
- Eckes, T., "Rater types in writing performance assessment : A classification approach to rater variability", Language Testing, 25, pp.155~185 (2008)
- He, T. H., "Multi-faceted Rasch measurement and bias patterns in EFL writing performance assessment", Psychological Reports, 112(2), pp.469~485 (2013)
- Hedgcock, J., "Collaborative oral", aural revision in foreign Language, 1992, pp.1(3), 255~276
- Kyung-AeJin, "Applying Standards-Based English Performance Assessment in Korean High Schools", 현대문법연구, (28), pp.231~244 (2002), 현대문법학회
- 김인숙, "Paper-based versus Image-based Scoring of Student Essays", 응용언어학, 27(2), pp.231~256 (2011), 한국응용언어학회
- Kondo-Brown, K., "A FACETS analysis of rater bias in measuring Japanese second language writing performance", Language Testing, 19, pp.3~31 (2002)
- Langer, A., "How writing shapes thinking" (1987), National Council of Teachers of English
- Lee, S. Y., "Validation of NES rater performance in the assessment of Korean university student English essays", doctor, pp.2013
- Leki, I., "A synthesis of research on second language writing" (2008), Routledge
- Lynch, B. K., "Using G-theory and many-facet Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants", Language Testing, 15(2), pp.158~180 (1998)
- McNamara, T. F., "Measuring second language performance" (1996), Longman
- 박리리, "A Comparative Analysis of Inter-rater Reliability in English Essay and Korean-English Translation Tests", 현대영어교육, 17(1), pp.27~48 (2016), 현대영어교육학회
- Rosenfeld, M., "The reading, writing, speaking, and listening tasks important for academic success at the undergraduate and graduate levels (TOEFL Monograph 21)" (2001), Educational Testing Service
- 신유선, "Investigating Variability in Tasks, Analytic Scoring Criteria, and Rater Groups on L2 Writing in English", Foreign Languages Education, 17(2), pp.27~57 (2010), 한국외국어교육학회
- Sternglass, M., "Time to know them: A longitudinal study of writing and learning at the college level" (1997), Erlbaum
- Weigle, S. C., "Effects of training on raters of ESL compositions: quantitative and qualitative approaches", doctor, pp.1994
- Weigle, S. C., "Using facets to model rater training effects", Language Testing, 15, pp.263~287 (1998)
- Weigle, S. C., "Assessing writing" (2002), Cambridge University Press