본문 바로가기
KERD
논문KCI 등재

한국어 말하기 평가의 발음 영역 채점에서의 채점자 특성에 따른 채점 경향 연구 -한국어 교육 경험과 전공을 중심으로-

Study of Raters' Rating Patterns on a Pronunciation Criteria in Speaking Assessment -Focusing on Rater Experience and Major-

1

조회

0

다운로드

25

인용

0

참고문헌

키워드

주제분류한국어와문학

초록

In second language performance assessment, raters variability has long been recognized as a potential threat to the validity of test takers' performance score. This study examines the effect of raters' rating patterns on a Pronunciation criteria in Speaking assessment. In this study, a focus was placed on the impact of raters' experience and majors on the judgements of Korean learners' pronunciation abilities. Twenty raters judged speech samples of 20 Korean language learners on six-point scale for accuracy(segmental,suprasegmental), fluency(speech rate and pause) and intelligibility respectively. As a Results Korean teachers who had more than one year experience had inter-rater reliability. However there were differences in leniency and severity. Also mirror image pattern was exhibited between the group with more than 5year-experience and the group with less than 5 year-experience. Moreover the analysis showed there were different rating patterns amongst three major groups: Korean Phonology, Korean literature and Korean Education as a foreign language. Such findings are expected to contribute to a better understanding of raters' scoring processes to the development of more reliable speaking assessment.

Abstract

In second language performance assessment, raters variability has long been recognized as a potential threat to the validity of test takers' performance score. This study examines the effect of raters' rating patterns on a Pronunciation criteria in Speaking assessment. In this study, a focus was placed on the impact of raters' experience and majors on the judgements of Korean learners' pronunciation abilities. Twenty raters judged speech samples of 20 Korean language learners on six-point scale for accuracy(segmental,suprasegmental), fluency(speech rate and pause) and intelligibility respectively. As a Results Korean teachers who had more than one year experience had inter-rater reliability. However there were differences in leniency and severity. Also mirror image pattern was exhibited between the group with more than 5year-experience and the group with less than 5 year-experience. Moreover the analysis showed there were different rating patterns amongst three major groups: Korean Phonology, Korean literature and Korean Education as a foreign language. Such findings are expected to contribute to a better understanding of raters' scoring processes to the development of more reliable speaking assessment.

참고문헌 (39)

  1. IBT 기반 말하기 평가를 위한 문항 유형개발, "한국어능력시험의 CBT", pp.2012, 박동호
  2. 신동일, "채점 경향 분석을 위한 Rasch 측정모형 적용 연구", 외국어교육, 8(1), pp.249~272 (2001), 한국외국어교육학회
  3. 신동일, "한국의 영어평가학2: 말하기 시험편" (2006), 한국 문화사
  4. 이향, "말하기 수행 평가에서 발음 범주 채점의 최적화 방안 연구 -일반화가능도 이론을 활용하여-", 한국어교육, 23(2), pp.301~329 (2012), 국제한국어교육학회
  5. 이향, "한국어 말하기 수행평가의 발음 범주 채점 방식에 따른 채점 신뢰도 분석 -다국면 라쉬 모형을 활용하여-", 외국어로서의 한국어교육, 37, pp.325~355 (2012), 연세대학교 언어연구교육원 한국어학당
  6. 이 향, "한국어 말하기 수행 평가의 발음 범주 채점에 대한 타당성 검증", 박사, 국어국문학과, pp.2013
  7. 이향, "한국어 말하기 평가의 발음 영역 채점에서의 채점자 특성에 따른채점 경향 연구 -한국어 교육 경험과 전공을 중심으로-", 2013년 제18차추계 전국학술대회 발표지, pp.127~140 (2013), 한국언어문화교육학회
  8. Bachman, L. F., "Investigating variability in tasks and rater judgements in a performance test of foreign language speaking", Language Testing, 12(2), pp.238~257 (1995)
  9. Barnwell, D., "Native native speakers and judge of oral proficiency in Spain", Language Testing, 6, pp.152~163 (1989)
  10. Carey, M. D., "Does a rater’s familiarity with a candidate’s pronunciation affect the rating in oral proficiency interviews?", Language Testing, 28(2), pp.201~219 (2011)[DOI]
  11. Chen, C., "Response style and cross-cultural comparisons of rating scales among East Asian and North American students", Psychological Science, 6, pp.170~175 (1995)
  12. Chalhoub-Deville, M., "Deriving oral assessment scales across different tests and rater groups", Language Testing, 12(1), pp.16~33 (1995)
  13. Brown, A., "The effect of rater variables in the development of an occupation-specific language performance test", Language Testing, 12(1), pp.1~15 (1995)
  14. Eckes, T., "Examining rater effects in TestDaF writing and speaking performance assessments: A many-facet Rasch analysis", Language Assessment Quarterly: An International Journal, 2(3), pp.197~221 (2005)
  15. Elder, C., "Bias in Language Assessment, In The encyclopedia of applied linguistics", pp.406~412 (2012), Wiley-Blackwell
  16. Hadden B. L., "Teacher and nonteacher perception of second language communication", Language Learning, 41, pp.1~24 (1991)
  17. Jones R. L., "Second Language performance testing: an overview, In Second Language performance testing", pp.15~24 (1985), University of Ottawa Press
  18. Levis, J. M., "Pronunciation and the assessment of spoken language, In Spoken English, applied linguistics and TESOL: Challenges for theory and practice", pp.245~270 (2008), Palgrave Macmillan
  19. Linacre, J. M., "Many-faceted Rasch measurement" (1989), MESA press
  20. Lunz, M. E., "Judge consistency and severity across grading periods", Evaluation & the Health Professions, 13(4), pp.425~444 (1990)
  21. Isaacs, T., "Rater experience, rating scale length, and judgments of L2 pronunciation: Revisiting research conventions", Language Assessment Quarterly, 10(2), pp.135~159 (2013)
  22. Jones, R. L., "Second language performance testing: An overview, In Second language performance testing", pp.15~24 (1985), University of Ottawa Press
  23. Language Testing, "Detecting DIF across thd different language group in a speaking test", Language Testing, 18, pp.89~114 (2001)
  24. 김현정, "An investigation of novice, developing, and experienced raters’ rating patterns on a second language speaking assessment.", 응용언어학, 26(4), pp.151~182 (2010), 한국응용언어학회
  25. Lumley, T., "Rater characteristics and rater bias: Implications for training", Language Testing, 12(1), pp.54~71 (1995)
  26. Lumley, T., "Perception of language -trained raters and occupational experts in a test of occupational English language proficiency", English for specific purpose, 17, pp.347~367 (1998)
  27. Lynch, B. K., "Using G-theory and many-facet Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants", Language Testing, 15(2), pp.158~180 (1998)
  28. McNamara, T., "Measuring second language performance" (1996), Longman
  29. Myford, C. M., "Detecting and Measuring Rater Effects Using Many-Facet Rasch Measurement: Part I", Journal of Applied Measurement, 4(4), pp.386~422 (2003)
  30. North, B., "The development of a common framework scale of language proficiency" (2000), Peter Lang
  31. North, B., "Scaling descriptors for language proficiency scales", Language Testing, 15(2), pp.217~262 (1998)
  32. Schaefer, E., "Rater bias patterns in an EFL writing assessment", Language Testing, 25(4), pp.465~493 (2008)[DOI]
  33. Shepard, L., "Comparison of procedures for detecting test-item bias with both internal and external ability criteria", Journal of Educational and Behavioral Statistics, 6(4), pp.317~375 (1981)
  34. Shohamy, E., "The effect of raters' background and training in the reliability of direct writing test", The Modern Language Journal, 76(1), pp.27~33 (1992)
  35. Thompson, I., "Foreign accent revisited: The English Pronunciation of Russian immigrants", Language Learning, 41, pp.177~204 (1991)
  36. Upshur, J. A., "Systematic effects in the rating of second-language speaking ability: Test method and learner discourse", Language Testing, 16(1), pp.82~111 (1999)
  37. Weigle, S. C., "Using FACETS to model rater training effects", Language Testing, 15(2), pp.263~287 (1998)
  38. Weigle, S. C., "Assessing Writing" (2002), Cambridge University Press
  39. Wigglesworth, G., "Exploring bias analysis as a tool for improving rater consistency in assessing oral interaction", Language Testing, 10(3), pp.305~319 (1993)

관련 논문AI추천

이 간행물의 다른 논문 (15건)