본문 바로가기
KERD
논문KCI 등재

A Comparative Analysis of Inter-rater Reliability in English Essay and Korean-English Translation Tests

2

조회

0

다운로드

2

인용

0

참고문헌

키워드

초록

Writing performance assessment is gaining momentum in the field of English education in Korea. Writing performance tests, however, are prone to subjective rating that may have adverse effects on the reliability of scores. Previous research recommended reinforced training and clearer scoring criteria. Rater training, however, is costly and demanding in the case of essay tests which require understanding and application of multiple criteria without much guidance on benchmark responses. This study thus sought to examine translation as an alternative writing task that may induce greater score reliability as well as accurate measurement of writing proficiency. Three raters scored the responses of 20 Korean college students to an English to Korean translation test and an English essay test solely based on the rubrics. The results showed that inter-rater reliability was higher for the translation test than for the essay test. In addition, the agreement between two of the raters reached a statistically significant level for the translation test. The implications of these findings are discussed in relation to the viability of implementing translation tests in secondary schools together with suggestions for further study on ways to strengthen the advantages of both essay and translation tests and make up for their weaknesses.

Abstract

Writing performance assessment is gaining momentum in the field of English education in Korea. Writing performance tests, however, are prone to subjective rating that may have adverse effects on the reliability of scores. Previous research recommended reinforced training and clearer scoring criteria. Rater training, however, is costly and demanding in the case of essay tests which require understanding and application of multiple criteria without much guidance on benchmark responses. This study thus sought to examine translation as an alternative writing task that may induce greater score reliability as well as accurate measurement of writing proficiency. Three raters scored the responses of 20 Korean college students to an English to Korean translation test and an English essay test solely based on the rubrics. The results showed that inter-rater reliability was higher for the translation test than for the essay test. In addition, the agreement between two of the raters reached a statistically significant level for the translation test. The implications of these findings are discussed in relation to the viability of implementing translation tests in secondary schools together with suggestions for further study on ways to strengthen the advantages of both essay and translation tests and make up for their weaknesses.

참고문헌 (48)

  1. Ahmed, A., "Improving marking quality through a taxonomy of mark schemes", Assessment in Education: Principles, Policy & Practice, 18(3), pp.259~278 (2011)
  2. 안경자, "Learning to Write Academic English: A Sociocultural Perspective on L2 Learners’ Writing Portfolios", 응용언어학, 25(2), pp.1~34 (2009), 한국응용언어학회
  3. Alderson, J., "Language test construction and evaluation" (2002), Cambridge University Press
  4. Bachman, L. F., "Fundamental considerations in language testing" (1990), Oxford University Press
  5. Bachman, L. F., "Language testing in practice" (1996), Oxford University Press
  6. Brown, A., "Interviewer variation and co-construction of speaking proficiency", Language Testing, 20, pp.1~25 (2003)
  7. Brown, G. T. L., "Tertiary assessment and higher education student outcomes: Policy, practice and research", pp.40~48 (2009), Ako Aotearoa
  8. 조동완, "번역 과제 발표 중심의 대학 영작문 수업에 관한 연구", Foreign Languages Education, 16(1), pp.261~289 (2009), 한국외국어교육학회
  9. Choi, Inn-Chull, (2010), SaramIn Publishing Co
  10. Choi, Inn-Chull, "Comparative analyses of computer-based ratings and human ratings based on performance testing of writing", Multimedia-Assisted Language Learning, 4(1), pp.165~184 (2001)
  11. 전유아, "Collocations in L2 Writing and Rater's Perceived Writing Proficiency", 응용언어학, 25(1), pp.101~129 (2009), 한국응용언어학회
  12. Methodological approaches to online scoring of essays, "Chung, G.", National Center for Research on Evaluation, Standards, and Student Testing, ~, pp.CSE Technical Report 461 (1997)
  13. Cohen, A. D., "Research on direct versus translated writing:Students’ strategies and their results", The Modern Language Journal, 85(2), pp.169~188 (2001)
  14. Cumming, A., "Writing expertise and second language proficiency", Language Learning, 39, pp.81~141 (1989)
  15. Eckes, T., "Rater types in writing performance assessments: A classification approach to rater variability", Language Testing, 25(2), pp.155~185 (2008)
  16. Eckes, T., "Operational rater types in writing assessment: Linking rater cognition to rater behavior", Language Assessment Quarterly, 9(3), pp.317~334 (2012)
  17. Enright, M. K., "Complementing human judgment of essays written by English learners with e-rater ® scoring", Language Testing, 27(3), pp.317~334 (2010)
  18. Fritz E., "Rater sensitivity to lexical accuracy, sophistication and range when assessing writing", Assessing Writing, 18(2), pp.173~181 (2013)
  19. Garcia, I., "Machine translation-assisted language learning: Writing for beginners", Computer Assisted Language Learning: An International Journal, 24(5), pp.471~487 (2011)
  20. Guo, L., "Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study", Assessing Writing, 18(3), pp.218~238 (2013)
  21. He, T. H., "Multi-faceted Rasch measurement and bias patterns in EFL writing performance assessment", Psychological Reports, 112(2), pp.469~485 (2013)
  22. Kyung-AeJin, "Applying Standards-Based English Performance Assessment in Korean High Schools", 현대문법연구, (28), pp.231~244 (2002), 현대문법학회
  23. Kane, M. T., "Validating measures of performance", Educational Measurement: Issues and Practice, 18(2), pp.5~17 (1999)
  24. 김인숙, "Paper-based versus Image-based Scoring of Student Essays", 응용언어학, 27(2), pp.231~256 (2011), 한국응용언어학회
  25. 김정아, "Grammatical and Discursive Features of Korean EFL Learners’ Direct, Translated and Back-translated Writing", 언어연구, 27(2), pp.373~392 (2010), 언어정보연구소[DOI]
  26. 김진아, "번역 품질평가에 관한 小考", 외국문학연구, (11), pp.4~124 (2002), 외국문학연구소
  27. 김연희, "쓰기 수행 평가의 채점 기준 변이성이 영작문 평가에 미치는 영향", 영어학, 13(3), pp.465~496 (2013), 한국영어학회[DOI]
  28. Kim, Young-Ran, "The use of L1 as a writing strategy in L2 writing tasks", GEMA Online Journal of Language Studies, 14(3), pp.33~50 (2014)
  29. 이상기, "Different Rater Groups and Their Feedback on Written Texts: Where Do They Focus Their Attention?", 영어교과교육, 12(1), pp.57~76 (2013), 한국영어교과교육학회[DOI]
  30. 이요안, "Reevaluating Advanced Speaking Proficiency for Korean Teachers of English: A Case Study", 응용언어학, 25(3), pp.303~336 (2009), 한국응용언어학회
  31. Lind Pantzare, A., "Interrater reliability in large-scale assessments: Can teachers score national tests reliably without external controls?", Practical Assessment, Research & Evaluation, 20(9) (2015)
  32. Lumley, T., "Assessment criteria in a large-scale writing test: What do they really mean to the raters?", Language Testing, 19(3), pp.246~276 (2002)
  33. Lumley, T., "Rater characteristics and rater bias: Implications for training", Language Testing, 12, pp.54~71 (1995)
  34. McNamara, T., "Measuring second language performance" (1996), Longman
  35. Nida, E. A., "The theory and practice of translation" (1982), Koninklijke Brill NV
  36. 박지선, "NEAT 말하기·쓰기 평가 도입에 대비한 고등학교 영어 말하기·쓰기 교수·학습 현황 분석", 현대영어교육, 13(2), pp.121~149 (2012), 현대영어교육학회[DOI]
  37. Park, Young-Soon, "The basic theory of interpretation and translation studies", Modern English Education, 3(2), pp.99~134 (2002)
  38. Plakans, L., "Independent vs. integrated writing tasks: A comparison of task representation", TESOL Quarterly, 44(1), pp.185~194 (2010)
  39. Schaefer, E., "Rater bias patterns in an EFL writing assessment", Language Testing, 25(4), pp.465~493 (2008)
  40. Shin, Bong-Soo, "Experimental study on the effects of Korean on English writing:Direct writing versus translation writing", Studies in Language, 14, pp.127~142 (1998)
  41. 신유선, "A FACETS Analysis of Rater Characteristics and Rater Bias in Measuring L2 Writing Performance", 영어어문교육, 16(1), pp.123~142 (2010), 한국영어어문교육학회
  42. 신유선, "Investigating Variability in Tasks, Analytic Scoring Criteria, and Rater Groups on L2 Writing in English", Foreign Languages Education, 17(2), pp.27~57 (2010), 한국외국어교육학회
  43. Tavakoli, M., "Direct versus translated writing: The effect of translation on learners’ second language writing ability", GEMA Online Journal of Language Studies, 14(2), pp.61~74 (2014)
  44. Uzawa, K., "Writing strategies in Japanese as a foreign language:Lowering or keeping up the standards", The Canadian Modern Language Review, 46, pp.178~194 (1989)
  45. Veerappan, V., "A review on IELTS writing test, its test results and inter rater reliability", Theory & Practice in Language Studies, 2(1), pp.138~143 (2012)
  46. Weigle, S., "Effects of training on raters of ESL compositions", Language Testing, 11(1), pp.197~223 (1994)
  47. Weigle, S., "Using FACETS to model rater training effects", Language Testing, 15, pp.263~287 (1998)
  48. Weigle, S. C., "Effects of task and rater background on the evaluation of ESL student writing: A pilot study", TESOL Quarterly, 37(2), pp.345~354 (2003)

관련 논문AI추천

이 간행물의 다른 논문 (10건)