영어 말하기 평가에서 나타난 한국인 교사 채점자의 채점 경향성 분석
An investigation into the rating tendency of Korean teachers as a non-native rater in a speaking assessment of English
1
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
The present study investigated the biases of Korean teachers who were non-native English speakers in rating test takers’ voice information, such as native-like pronunciation. To examine the potential biases, two different scoring methods were adopted. One was a normal scoring method where five qualified Korean teachers assessed 100 test takers’ responses while listening to their voice files. For the other method, the original voice files were transcribed into text files. After a one-year time interval, the same raters reassessed the text files without the test takers’ voice information. The results for the three common rating domains of “task completion,” “discourse organization,” and “language use” significantly different between the two scoring methods. Also, “pronunciation” and “fluency” were more highly correlated with the three domains in the first rating than in the second rating. That is, even when the raters were asked to assess the spoken language of test takers by analytic scoring, ratings for “pronunciation” and “fluency” seemed to be strongly associated with the scoring domains when taking into account the test takers’ voice information. The results were confirmed by interviews with the raters. Also, as expected, four of the five raters scored test takers more severely in the text-based rating. Pedagogical implications are suggested to overcome biases that non-native speaking teacher raters may have in judging oral English performance.
Abstract
The present study investigated the biases of Korean teachers who were non-native English speakers in rating test takers’ voice information, such as native-like pronunciation. To examine the potential biases, two different scoring methods were adopted. One was a normal scoring method where five qualified Korean teachers assessed 100 test takers’ responses while listening to their voice files. For the other method, the original voice files were transcribed into text files. After a one-year time interval, the same raters reassessed the text files without the test takers’ voice information. The results for the three common rating domains of “task completion,” “discourse organization,” and “language use” significantly different between the two scoring methods. Also, “pronunciation” and “fluency” were more highly correlated with the three domains in the first rating than in the second rating. That is, even when the raters were asked to assess the spoken language of test takers by analytic scoring, ratings for “pronunciation” and “fluency” seemed to be strongly associated with the scoring domains when taking into account the test takers’ voice information. The results were confirmed by interviews with the raters. Also, as expected, four of the five raters scored test takers more severely in the text-based rating. Pedagogical implications are suggested to overcome biases that non-native speaking teacher raters may have in judging oral English performance.
참고문헌 (48)
- 교육과학기술부, "초등학교 교육과정 해설 외국어(영어)" (2008), 한솔기획
- "2013년 국가영어능력평가시험(2․3급) 시행계획 발표.조간보도자료" (2013), 교육부[URL]
- 신동일, "영어말하기평가의 문항 분석: 의사소통 기능을 중심으로", 영어교육, 64(2), pp.103~124 (2009), 한국영어교육학회
- Adams, R. J., "ConQuest 3.0", ACER, ~ (2012)
- Anderson-Hsieh, J., "The relationship between native speaker judgments of nonnative pronunciation and deviance in segmentals, prosody, and syllable structure", Language Learning, 42, pp.529~555 (1992)
- Brooks, R. I, "Comparing native and non-native raters of us federal government speaking tests", Doctor, pp.2013
- Brown, A., "The effect of rater variables in the development of an occupation-specific language performance test", Language Testing, 12(1), pp.1~15 (1995)
- Canagarajah, S., "Changing communicative needs, revised assessment objectives:Testing English as an International Language", Language Assessment Quarterly, 3(3), pp.229~242 (2006)
- Carey, M. D., "Does a rater’s familiarity with a candidate’s pronunciation affect the rating in oral proficiency interviews?", Language Testing, 28(2), pp.201~219 (2011)
- Crystal D, "English as a global language" (1997), Cambridge University Press
- Derwing, T. M., "Accent, intelligibility, and comprehensibility", Studies in Second Language Acquisition, 19, pp.1~16 (1997)
- Dimova, S., "Reduction in language testing", Proceedings of the SJUSK conference on contemporary speech habits, 2, pp.41~58 (2013)
- Elder, C., "Assessing English as a lingua franca", Annual Review of Applied Linguistics, 26(1), pp.232~301 (2006)
- ETS, "Test of spoken English: Standard-setting manual", ETS, ~ (1995)
- "TOEFL iBT test—Integrated speaking rubrics (scoring standards)" (2008)[URL]
- Fleiss, J. L., "Statistical methods for rates and proportions" (1981), John Wiley & Sons
- Graddol D, "The future of English?: A guide to forecasting the popularity of the English language in the 21st century" (1997), British Council
- Hamp-Lyons L, "Research perspectives on English for academic purposes", pp.101~116 (2001), Cambridge University Press
- Rater effects in ITA testing: ESL teachers’ versus American undergraduates’ judgments of accentedness, comprehensibility, and oral proficiency, "Hsieh, C. N.", 47~74, pp.Spaan Fellow Working Papers in Second or Foreign Language Assessment, 9 (2011)
- "IELTS Speaking band descriptors (public version)" (2017)[URL]
- Isaacs, T., "Towards defining a valid assessment criterion of pronunciation proficiency in non-native English speaking graduate students", Canadian Modern Language Review, 64(4), pp.555~580 (2008)
- Issacs, T., "Phonological memory, attention control, and musical ability : Effects of individual differences on rater judgements of second language speech", Applied Psycholinguistics, 32, pp.113~140 (2011)
- Jenkins, J., "The phonology of English as an international language" (2000), Oxford University Press
- Jenkins, J., "A sociolinguistically based, empirically researched pronunciation syllabus for English as an international language", Applied Linguistics, 23(1), pp.83~103 (2002)
- Jenkins, J., "The spread of EIL: A testing time for testers", ELT Journal, 60(1), pp.42~50 (2006)
- Ratings of L2 oral performance in English: Relative impact of rater characteristics and acoustic measures of accentedness, "Kang, O.", 181~205, pp.Spaan Fellow Working Papers in Second or Foreign Language Assessment, 6(1) (2008)
- Kang, O., "Impact of rater characteristics and prosodic features of speaker accentedness on ratings of international teaching assistants’ oral performance", Language Assessment Quarterly, 9(3), pp.249~269 (2012)
- Kang O., "Assessment in second language pronunciation" (2017), Routledge
- Kim, Y., "An investigation into native and non-native teachers’ judgments of oral English performance: A mixed methods approach", Language Testing, 26(2), pp.187~217 (2009)
- Kling, J., "Fremdsprachen in Studium und Lehre - Chancen und Herausforderungen für den Wissenserwerb", pp.247~268 (2015), Peter Language International Academic Publishers
- Lazaraton, A., "Multilingualism and assessment: Achieving transparency, assuring quality, sustaining diversity-proceedings of the ALTE Berlin conference", pp.296~309 (2005), Cambridge University Press
- Lowenberg, p., "Assessing English profiency in the expanding circle. World Englishes: Whose standards?", World Englishes, 12(1), pp.95~106 (2002)
- Ma, R., "The Role of pronunciation in speaking test ratings", Master, pp.2015
- Munro, M. J., "Phonology and second language acquisition", pp.193~218 (2008), John Benjamins
- Munro, M. J., "Foreign accent, comprehensibility, and intelligibility in the speech of second language learners", Language Learning, 45(1), pp.73~97 (1995)
- Munro, M. J., "Modelling perceptions of the comprehensibility and accentedness of L2 speech : The role of speaking rate", Studies in Second Language Acquisition, 23, pp.451~468 (2001)
- Ockey, G. J., "From one to multiple accents on a test of L2 listening comprehension", Applied Linguistics, 37(5), pp.693~715 (2016)
- Seidelhofer, b., "Closing the conceptual gap : The case for a description of English as a lingua franca", International Journal of Applied Linguistics, 11, pp.133~158 (2001)
- Shi, L., "Native-and nonnative-speaking EFL teachers’ evaluation of Chinese students’ English writing", Language Testing, 18, pp.303~325 (2001)
- Weir, C. J., "Measured construct : A history of Cambridge English language examinations 1913-2012", Cambridge English : Research Notes, 51, pp.2~6 (2013)
- Wennerstrom, A., "Perspectives on fluency", pp.102~127 (2000), University of Michigan Press
- Winke, P., "The relationship between raters’ prior language study and the evaluation of foreign language speech samples" (2011), Educational Testing Service
- Winke, P., "Raters’ L2 background as a potential source of bias in rating oral performance", Language Testing, 30(2), pp.231~252 (2013)
- How do raters from India perform in scoring the TOEFL iBT speaking section and what kind of training helps?, "Xi, X.", Educational Testing Service, ~, pp.TOEFL iBT Research Report No. RR-09-31 (2009)
- Yano, Y., "World Englishes in 2000 and beyond", World Englishes, 20(2), pp.119~132 (2001)
- Zhang, Y., "Judgments of oral proficiency by non-native and native English speaking teacher raters: Competing or complementary constructs?", Language Testing, 28(1), pp.31~50 (2011)