고등학생 영어학습자의 말하기 평가에서 채점자 훈련의 효과 및 채점자 인식 분석
An analysis of rater perceptions and the effectiveness of the rater training in speaking assessment
1
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
This study aims to examine the effectiveness of rater training as to the rater reliability in evaluating speaking skill, and to analyze how teachers’ perceptions changed as the training proceeded. For the research, four English teachers in Korean high school rated 15 students four times, once a week, and were interviewed after each rating period. The scoring data was analyzed through the ManyFacet Rash Measurement (MFRM) and the interviews were also analyzed. The results are as follows. First, both the inter- and intra-rater reliability increased. Although four raters were all leniently biased from the first session, the difference of the degree of leniency was closely narrowed down. Also, misfit aspect observed from one of the raters disappeared as the biases incurred by the lack of understanding of the scoring criteria reduced. Second, contrary to their previous concerns about the result of the training, teachers agreed on the need for the training and its effectiveness. The interview revealed that the training encouraged teachers to have confidence in their expertise, eventually raising reliability in their rating. The study demonstrates how rater training can establish rater reliability and most importantly implies that it can be successfully settled in Korean high school.
Abstract
This study aims to examine the effectiveness of rater training as to the rater reliability in evaluating speaking skill, and to analyze how teachers’ perceptions changed as the training proceeded. For the research, four English teachers in Korean high school rated 15 students four times, once a week, and were interviewed after each rating period. The scoring data was analyzed through the ManyFacet Rash Measurement (MFRM) and the interviews were also analyzed. The results are as follows. First, both the inter- and intra-rater reliability increased. Although four raters were all leniently biased from the first session, the difference of the degree of leniency was closely narrowed down. Also, misfit aspect observed from one of the raters disappeared as the biases incurred by the lack of understanding of the scoring criteria reduced. Second, contrary to their previous concerns about the result of the training, teachers agreed on the need for the training and its effectiveness. The interview revealed that the training encouraged teachers to have confidence in their expertise, eventually raising reliability in their rating. The study demonstrates how rater training can establish rater reliability and most importantly implies that it can be successfully settled in Korean high school.
참고문헌 (59)
- 강순부, "고전검사이론과 문항반응이론을 이용한 사관학교 1차 선발 수학시험의 문항분석", 공사논문집, 67(1), pp.113~128 (2016)
- 교육부, "서울특별시 고등학교 교육 과정 편성 지침" (2008), 교육부
- 교육부, "2기 수업 혁신 방안" (2018), 교육부
- 김진의, "Rasch 평정척도 모형을 적용한 중학생 체육수업인식 측정척도 개발 및 탐색", 한국체육과학회지, 27(1), pp.841~852 (2018), 한국체육과학회[DOI]
- 백현영, "중학교 영어교사의 말하기평가 채점경향 분석", 언어과학, 18(4), pp.77~99 (2011), 한국언어과학회
- 이창수, "한국 중· 고등학교 교사들의 영어작문 채점 훈련에서 나타나는채점자 편향과 그 원인", 박사, pp.2014
- 장소영, "언어교육평가 연구를 위한 FACETS 프로그램" (2009), 글로벌콘텐츠
- 조재윤, "일반화가능도 이론을 이용한 쓰기 평가의 오차원 분석 및 신뢰도 추정 연구", 국어교육, (128), pp.325~358 (2009), 한국어교육학회
- 최연희, "채점자 훈련이 영어 작문 채점에 미치는 효과에 대한 연구: FACETS 분석을 통한 신뢰도 변화 분석", 응용언어학, 18(1), pp.13~292 (2002), 한국응용언어학회
- 최윤곤, "한국어 교사의 채점자 훈련 모형 개발", 한국언어문화학, 10(2), pp.295~316 (2013), 국제한국언어문화학회
- 한문섭, "SEPT 채점 신뢰도와 타당도 연구", 영어교육연구, 16(2), pp.285~299 (2004), 팬코리아영어교육학회
- 황명환, "영어작문수업에서의 자동작문채점기 FES 활용에 대한타당성 연구", 영어교육연구, 27(3), pp.261~283 (2015), 팬코리아영어교육학회[DOI]
- Bijani, H, "Investigating the validity of oral assessment rater training program: A mixed-methods study of raters’ perceptions and attitudes before and after training", Cogent Education, 5(1) (2018)
- Brown, A, "The effect of rater variables in the development of an occupation-specific language performance test", Language Testing, 12(1), pp.1~15 (1995)
- Brown, A, "Interviewer variation and the co-construction of speaking proficiency", Language Testing, 20(1), pp.1~25 (2003)
- Canale, N, "Theoretical Bases of Communicative Approaches to Second Language Teaching and Testing", Applied Linguistics, 1, pp.1~47 (1980)
- Chalhoub‐Deville, M, "Rater judgment and English language speaking proficiency", World Englishes, 24(3), pp.383~391 (2005)
- Cooper, W. H, "Ubiquitous halo", Psychological bulletin, 90(2), pp.218~244 (1981)
- Corbin, J, "Basics of qualitative research: Techniques and procedures for developing grounded theory" (2009), Sage Publications
- Cumming, A, "Expertise in evaluating second language compositions", Language Testing, 7(1), pp.31~51 (1990)
- Davis. L, "The influence of interlocutor proficiency in a paired oral assessment", Language Testing, 26(3), pp.367~398 (2009)
- Davis, L, "The influence of training and experience on rater performance in scoring spoken language", Language Testing, 33(1), pp.117~335 (2016)
- Eckes, T, "Introduction to many-facet Rasch measurement: Analyzing and evaluating rater-mediated assessments" (2011), Lang
- Elder, C, "Individual feedback to enhance rater training : Does it work?", Language Assessment Quarterly : An International Journal, 2(3), pp.175~196 (2005)
- Engelhard Jr, G, "Examining rater errors in the assessment of written composition with a many‐faceted Rasch model", Journal of Educational Measurement, 31(2), pp.93~112 (1994)
- Fayer, J. M, "Native and nonnative judgments of intelligibility and irritation", Language Learning, 37(3), pp.313~326 (1987)
- Fulcher, G, "Effective rating scale development for speaking test performance decision trees", Language Testing, 28(1), pp.5~29 (2011)
- Herman, J. L, "A practical guide to alternative assessment" (1992), Association for supervision and curriculum Development
- Hill, K, "Who should be the judge? The use of non-native speakers as raters on a test of English as an international language", Melbourne Papers in Language Testing, 5(2), pp.29~50 (1996)
- Hughes, A, "Testing for language teachers" (1989), Cambridge University Press
- Johnson, J. S, "The influence of rater language background on writing performance assessment", Language Testing, 26(4), pp.485~505 (2009)
- Khabbazbashi, N, "Topic and background knowledge effects on performance in speaking asessment", Language Testing, 34(1), pp.23~48 (2017)
- Kim, H. J, "A qualitative analysis of rater behavior on an L2 speaking assessment", Language Assessment Quarterly, 12(3), pp.239~261 (2015)
- 김현정, "An investigation of novice, developing, and experienced raters’ rating patterns on a second language speaking assessment.", 응용언어학, 26(4), pp.151~182 (2010), 한국응용언어학회
- Kim, S, "A comparative analysis of the ratings in performance assessment using generalizability theory and many-facet Rasch model", Journal of Applied Measurement, 10(4), pp.408~423 (2009)
- Kim, Y, "An investigation into native and non-native teachers’ judgments of oral English performance : A mixed methods approach", Language Testing, 26(2), pp.187~217 (2009)
- Knight, F. B, "The effect of the"acquaintance factor"upon personal judgments", Journal of Educational Psychology, 14, pp.129~142 (1923)
- Knoch, U, "Investigating the effectiveness of individualized feedback to rating behavior : A longitudinal study", Language Testing, 28(2), pp.179~200 (2011)
- Kwon, "English teachers’ concerns about the speaking", writing tests of the National English Ability Test, 2010, pp.13, 64~88
- Lim, G. S, "The development and maintenance of rating quality in performance writing assessment : A longitudinal study of new and experienced raters", Language Testing, 28(4), pp.543~560 (2011)
- Linacre, J. M, "Many-facet Rasch Measurement" (1989), MESA Press
- Lumley, T, "Rater characteristics and rater bias : Implications for training", Language Testing, 12(1), pp.54~71 (1995)
- Lunz, M. E, "Judge consistency and severity across grading periods", Evaluation & the Health Professions, 13(4), pp.425~444 (1990)
- Meiron, B. E, "Rating oral proficiency tests: A triangulated study of rater thought processes" (1998), Language Testing Research Colloquium
- Miles, M. B, "Qualitative data analysis: An expanded sourcebook" (1994), Sage
- Nakatsuhara, F, "Effect of test-taker characteristics and the number of participants in group oral tests", Language Testing, 28(4), pp.483~508 (2011)
- Orr, M, "The FCE Speaking test : Using rater reports to help interpret test scores", System, 30, pp.143~154 (2002)
- Papajohn, D, "Concept mapping for rater training", TESOL Quarterly, 36, pp.219~333 (2002)
- Saal, F. E, "Rating the ratings : Assessing the psychometric quality of rating data", Psychological bulletin, 88(2), pp.413~ (1980)
- Scarcella, R. C, "The Tepestry of language learning: The individual in the communicative classroom" (1992), Heinle & Heinle
- Schaefer, E, "Rater bias patterns in an EFL writing assessment", Language Testing, 25(4), pp.465~493 (2008)
- Shi, L, "Native-and nonnative-speaking EFL teachers’ evaluation of Chinese students’ English writing", Language Testing, 18, pp.303~325 (2001)
- 신유선, "A FACETS Analysis of Rater Characteristics and Rater Bias in Measuring L2 Writing Performance", 영어어문교육, 16(1), pp.123~142 (2010), 한국영어어문교육학회
- Van Moere, A, "A psycholinguistic approach to oral language assessment", Language Testing, 29(3), pp.325~344 (2012)
- Weigle, S. C, "Using FACETS to model rater training effects", Language Testing, 15(2), pp.263~287 (1998)
- Weigle, S. C, "Assessing writing" (2002), Cambridge University Press
- Wigglesworth, G, "Exploring bias analysis as a tool for improving rater consistency in assessing oral interaction", Language Testing, 10(3), pp.305~319 (1993)
- Xi, X, "How do raters from India perform in scoring the TOEFL iBT speaking section and what kind of training helps", ETS Research Report Series, 2009(2), pp.i~37 (2009)
- 이지연, "영작문 평가를 위한 채점자 훈련의 방향", 현대영어교육, 10(3), pp.217~241 (2009), 현대영어교육학회