Investigation of Raters' Behavior on Pronunciation Ratings in an English Speaking Test
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
The purpose of this study is to examine how consistently raters evaluate the pronunciation of native and nonnative English-speaking oral performance. Eleven native English- speaking raters participated in scoring twelve test-takers; eight of the test-takers were Koreans with various proficiency levels and one African- American, one British, one South African and one Filipino. The quantitative results showed that the raters showed variations in severity of pronunciation judgments especially for the African-American, the South African and one Korean with a high level of oral proficiency. The verbal protocol data revealed the following potential factors of the rater variation on pronunciation ratings: (1) raters’ different perceptions regarding standard English, (2) raters’ different degrees of familiarity with test-takers’ accents, (3) raters' utilization of criterion-irrelevant features, (4) testing method effects and (5) raters' application of their own scoring criteria. The notions of native accents and construct validity issues for the pronunciation ratings are discussed.
Abstract
The purpose of this study is to examine how consistently raters evaluate the pronunciation of native and nonnative English-speaking oral performance. Eleven native English- speaking raters participated in scoring twelve test-takers; eight of the test-takers were Koreans with various proficiency levels and one African- American, one British, one South African and one Filipino. The quantitative results showed that the raters showed variations in severity of pronunciation judgments especially for the African-American, the South African and one Korean with a high level of oral proficiency. The verbal protocol data revealed the following potential factors of the rater variation on pronunciation ratings: (1) raters’ different perceptions regarding standard English, (2) raters’ different degrees of familiarity with test-takers’ accents, (3) raters' utilization of criterion-irrelevant features, (4) testing method effects and (5) raters' application of their own scoring criteria. The notions of native accents and construct validity issues for the pronunciation ratings are discussed.
참고문헌 (37)
- Alderson, J. C, "Language Test Construction and Evaluation" (1995), Cambridge University Press
- Bachman, L. F, "Fundamental Considerations in Language Testing" (1990), Oxford University Press
- Bachman, L. F, "Investigating Variability in Tasks and Rater Judgments in a Performance Test of Foreign Language Speaking", Language Testing, 12, pp.239~257 (1995)
- Bachman, L. F, "The Construct Validation of Some Components of Communicative Proficiency", TESOL Quarterly, 16(4), pp.449~465 (1982)
- Bachman, L. F, "The Evaluation of Communicative Language Proficiency: A Critique of the ACTFL Oral Interview", The Modern Language Journal, 70(4), pp.380~390 (1986)
- Bent, T, "The Interlanguage Speech Intelligibility Benefit", Journal of the Acoustical Society of America, 114(3), pp.1600~1610 (2003)
- Bonk, W. J, "A Many-Facet Rasch Analysis of the Second Language Group Oral Discussion Task", Language Testing, 20(1), pp.89~110 (2003)
- Brennan, R. L, "Generalizability Theory" (2001), Springer
- Brown, A, "The Effect of Rater Variables in the Development of an Occupation Specific Language Performance Test", Language Testing, 12(1), pp.277~303 (1995)
- Celce-Murcia, M, "Teaching Pronunciation: A Reference for Teachers of English to Speakers of Other Language" (1996), Cambridge University Press
- Chalhoub-Deville, M, "Rater Judgment and English Language Speaking Proficiency", World Englishes, 24(3), pp.383~391 (2005)
- Connor-Linton, J, "Looking Behind the Curtain: What do L2 Composition Ratings Really Mean?", TESOL Quarterly, 29(4), pp.792~796 (1995)
- de Jong, W. N, "On Validating a Pronunciation Test", ELT Journal, 31, pp.233~239 (1977)
- Davies, A, "The Native Speaker: Myth and Reality" (2003), Multilingual Matters
- Dobbyn, M, "An Objective Test of Pronunciation for Large Classes", ELT Journal, 30, pp.242~244 (1976)
- Elliott, A. R, "On the Teaching and Acquisition of Pronunciation within Communicative Approach", Hispania, 80(1), pp.95~108 (1997)
- ESPT Score Report, "ESPT", ESPT Academy, ~ (2010)
- Flowerdew, J, "Research of Relevant to Second Language Lecture Comprehension - An Overview, In Academic Listening: Research Perspectives", pp.7~29 (1994), Cambridge University Press
- Green, A, "Verbal Protocol Analysis in Language Testing Research: A Handbook" (1998), Cambridge University Press
- Hendricks, D, "Oral Proficiency Testing in an Intensive English Language Program, In Research in Language Testing, Rowley", pp.77~90 (1980), Newbury House Publishers
- Jenkins, J, "Which Pronunciation Norms and Models for English as an International Language?", ELT Journal, 52, pp.119~126 (1998)
- Kachru, B. B, "Englishization and Contact Linguistics", World Englishes, 13(2), pp.134~154 (1994)
- KICE, "Public Hearing on National English Ability Test Developments and Administrations: Levels 2 and 3" (2010), Korea Institute for Curriculum and Evaluation
- Kim, Hyun-Ju, "Rater Reliability in L2 Oral Proficiency Tests", 영어교육, 61(3), pp.105~118 (2006), 한국영어교육학회
- Koren, S, "Foreign Language Pronunciation Testing: A New Approach", System, 23, pp.387~400 (1995)
- Lowenberg, P, "Assessing English Proficiency in the Expanding Circle", World Englishes, 12(3), pp.431~435 (2002)
- Lynch, B. K, "Using G Theory and Many-facet Rasch Measurement in the Development of Performance Assessments of the ESL Speaking Skills of Immigrants", Language Testing, 15(2), pp.158~180 (1998)
- Lumley, T, "Rater Characteristics and Rater Bias: Implications for Training", Language Testing, 12(1), pp.55~71 (1995)
- Munro, M. J, "Foreign Accent, Comprehensibility and Intelligibility in the Speech of Second Language learners", Language Learning, 4, pp.73~97 (1995)
- Murphy, J. M, "Oral Communication in TESOL: Integrating Speaking, Listening, and Pronunciation", TESOL Quarterly, 25(1), pp.51~75 (1991)
- Orr, M, "The FCE Speaking Test: Using Rater Reports to Help Interpret Test Scores", System, 30, pp.143~154 (2002)
- Parish, C, "A Practical Philosophy of Pronunciation", TESOL Quarterly, 11, pp.311~317 (1977)
- Richards, J. C, "Listening Comprehension: Approach, Design, Procedure", TESOL Quarterly, 17, pp.219~239 (1983)
- Shohamy, E, "The Stability of Oral Proficiency Assessment on the Oral Interview Testing Procedure", Language Learning, 33, pp.527~540 (1983)
- Vaughan, C, "Holistic Assessment: What Goes on in the Rater’s Mind?, In Assessing Second Language Writing in Academic Contexts", pp.111~125 (1991), Ablex
- Van Weeren, J., "Testing Pronunciation: An Application of Generalizability Theory", Language Learning, 37, pp.109~122 (1987)
- Weigle, S. C, "Effects of Training on Raters of ESL Compositions", Language Testing, 11, pp.197~223 (1994)