본문 바로가기
KERD
논문KCI 등재

A Study of the Korean English Teachers’ Rating Tendency to the Oral English Proficiency Tests

2

조회

0

다운로드

3

인용

0

참고문헌

키워드

주제분류영어와문학

초록

The paper investigates the Korean teachers’ scoring patterns and interpretation of oral English proficiency. More specifically, this study examines their evaluation patterns with regard to internal consistency and severity over various tasks and criteria. Twenty-two raters (11 natives, 11 non-natives) took part in rating English speeches of fourteen subjects based on four tasks (open-ended response, directed response, picture-cued description, and opinion), using five criteria: fluency, pronunciation, grammar, vocabulary, and discourse. The raters were also asked to provide open-ended written comments on the factors contributing to their scoring. Results yielded by both quantitative (MFRM) and qualitative analyses of rater data revealed that most of the Korean teachers maintained an acceptable level of internal consistency and severity patterns. However, they exhibited some distinctive characteristics different from their native counterparts for some criteria: stricter grading for grammar accuracy and lenient measurements for discourse cohesion. Furthermore, they have rated differently in terms of severity across some tasks. Among the tasks measured, they exhibited strict grading for the opinion task in which their native counterparts showed lenient grading. The analysis of the comments on a questionnaire revealed that the variations resulted from raters’ different perspectives on the L2 speaking assessment. The present study enriches our understanding of Korean teachers’ scoring patterns on oral English proficiency.

Abstract

The paper investigates the Korean teachers’ scoring patterns and interpretation of oral English proficiency. More specifically, this study examines their evaluation patterns with regard to internal consistency and severity over various tasks and criteria. Twenty-two raters (11 natives, 11 non-natives) took part in rating English speeches of fourteen subjects based on four tasks (open-ended response, directed response, picture-cued description, and opinion), using five criteria: fluency, pronunciation, grammar, vocabulary, and discourse. The raters were also asked to provide open-ended written comments on the factors contributing to their scoring. Results yielded by both quantitative (MFRM) and qualitative analyses of rater data revealed that most of the Korean teachers maintained an acceptable level of internal consistency and severity patterns. However, they exhibited some distinctive characteristics different from their native counterparts for some criteria: stricter grading for grammar accuracy and lenient measurements for discourse cohesion. Furthermore, they have rated differently in terms of severity across some tasks. Among the tasks measured, they exhibited strict grading for the opinion task in which their native counterparts showed lenient grading. The analysis of the comments on a questionnaire revealed that the variations resulted from raters’ different perspectives on the L2 speaking assessment. The present study enriches our understanding of Korean teachers’ scoring patterns on oral English proficiency.

참고문헌 (41)

  1. Bachman, L. F., "Fundamental considerations in language testing" (1990), Oxford University Press
  2. Bachman, L. F., "Statistical analyses for language assessment" (2004), Cambridge University Press
  3. Bachman, L. F., "Language testing in practice" (1996), Oxford University Press
  4. Barnwell, D., "Naive native speakers and judgments of oral proficiency in Spanish", Language Testing, 6, pp.152~163 (1989)
  5. Bridgeman, B., "TOEFL iBT speaking test scores as indicators of oral communicative language proficiency", Language Testing, 29(1), pp.91~108 (2011)
  6. Brown, A., "The effect of rater variables in the development of an occupation-specific language performance test", Language Testing, 12, pp.1~15 (1995)
  7. Brown, A., "Getting the balance right: Criteria in integrated speaking tasks", Paper presented at the 24th Language Testing Research Colloquium (2002)
  8. Brutt-Griffler, J., "Transcending the nativeness paradigm", World Englishes, 20(1), pp.99~106 (2001)
  9. Caracelli, V. J., "Data analysis strategies for mixed-method evaluation designs", Educational Evaluation and Policy Analysis, 15, pp.195~207 (1993)
  10. Clark, V. L. P., "The mixed methods reader" (2008), Sage
  11. Crosthwaite, P., "The effect of collaboration on the cohesion coherence of L2 narrative discourse between English NS and Korean L2 English users", Asian EFL Journal, 13(4), pp.135~166 (2010)
  12. Crystal, D., "English as a global language" (2003), Cambridge University Press
  13. Fayer, J. M., "Native and nonnative judgements of intelligibility and irritation", Language Learning, 37(3), pp.313~326 (1987)
  14. Galloway, V. B., "Perceptions of the communicative efforts of American students of Spanish", Modern Language Journal, 64, pp.428~433 (1980)
  15. Givón, T., "Coherence in spontaneous text", pp.59~115 (1995), Benjamins
  16. Hill, K., "Who should be the judge? The use of non-native speakers as raters on a test of of English as an international language", Current developments and alternatives in language assessment: Proceedings of LTRC 96 (1997), University of Jyväskylä and University of Tampere
  17. Hinkel, E., "Native and non-native speakers’ pragmatic interpretations of English texts", TESOL Quarterly, 28(2), pp.353~376 (1994)
  18. Hymes, D., "Sociolingusitics: Selected readings" (1972), Penguin
  19. Ivankova, N., "Qualitative research in applied linguistics: A practical introduction", pp.135~164 (2009), CPU
  20. Kachchaf, R., "Rater language backgrounds as a source of measurement error in the testing of English language learners", Applied Measurement in Education, 25, pp.167~177 (2013)
  21. 강석한, "A Comparative Study on Criteria and Tasks in Korean English Speaking Assessment by Native and Non-native Raters", 어학연구, 48(2), pp.241~262 (2012), 언어교육원
  22. Kobayashi, T., "Native and nonnative reactions to ESL compositions", TESOL Quarterly, 26(1), pp.81~112 (1992)
  23. 이춘희, "Improving inter-rater reliability in oral proficiency test at college level", 현대영어영문학, 54(1), pp.367~387 (2010), 한국현대영어영문학회
  24. Lowenberg, P. H., "Assessing English proficiency in the expanding circle", World Englishes, 21, pp.431~435 (2002)
  25. Lumley, T. &, "Rater characteristics and rater bias : Implication for training", Language Testing, 12, pp.54~71 (1995)
  26. Luoma, S., "Assessing speaking" (2004), Cambridge University Press
  27. Melchers, G., "World Englishes" (2003), Arnold
  28. Myford, C. M., "Introduction to Rasch measurement", pp.386~422 (2004), JAM Press
  29. North, B., "Scaling descriptors for language proficiency scales", Language Testing, 15(2), pp.217~262 (1998)
  30. Pennycook, A., "Global Englishes, rip slime, and performativity", Journal of Sociolinguistics, 7(4), pp.513~533 (2003)
  31. Santos, T., "Professor’s reactions to the academic writing of non-native speaking students", TESOL Quarterly, 22(1), pp.69~90 (1989)
  32. Shohamy, E., "The effect of raters’ background and training on the reliability of direct writing tests", The Modern Language Journal, 76(1), pp.27~33 (1992)
  33. Smith, L. E., "The comprehensibility of three varieties of English for college students in seven countries", Language Learning, 32, pp.238~272 (1982)
  34. Taylor, L. B., "The changing landscape of English : Implications for language assessment", ELT Journal, 60, pp.51~60 (2006)
  35. Weigle, S. C., "Using FACETS to model rater training effects", Language Testing, 15, pp.263~287 (1998)
  36. Winke, P., "Raters’ L2 background as a potential source of bias in rating oral performance", Language Testing, 30(2), pp.231~252 (2012)
  37. Wright, B. D., "Reasonable mean-square fit values", Rasch Measurement Transactions SIG, 8, pp.370~ (1994)
  38. Zhang, Y., "Judgements of oral proficiency by non-native and native English speaking teacher raters: Competing or complementary constructs?", Language Testing, 28(1), pp.31~50 (2011)
  39. Yuan, Y., "An inquiry into empirical pragmatics data-gathering methods : Written DCTs, oral DCTs, field notes, and natural conversations", Journal of Pragmatics, 33(2), pp.271~292 (2001)
  40. 유경아, "The effect of raters’ language background on English-speaking test ratings across test-takers’ oral proficiency levels", 응용언어학, 26(4), pp.395~419 (2010), 한국응용언어학회

관련 논문AI추천

이 간행물의 다른 논문 (16건)