영어 말하기 평가의 채점자 신뢰성과 편향성 조사: 구술 인터뷰, 컴퓨터 구술시험, 버슨트의 비교를 통하여
Examining rater reliability and bias in measuring English speaking performance: Through a comparison of scores on an oral interview, a computerized oral test and a Versant
2
조회
0
다운로드
13
인용
0
참고문헌
키워드
초록
The purpose of this study was to investigate inter- and intra- rater reliability in aninterview and a computerized oral test. It was also examined whether ratercharacteristics influenced on their reliability and biases, and finally the scores of bothtests were compared with those of the Versant test using an automated computer ratingsystem. For the study, the data from 21 Korean university students and 18 Korean ornative speakers of English raters with various characteristics were collected. Some ofthe main findings from the study were as follows. First, rater severity was significantlydifferent in each test, but each rater consistently graded on both tests suggesting lowerinter-rater reliability and higher intra-rater reliability. Secondly, rater severity wasimpacted by the rater characteristics such as mother tongue, gender, age, and major. Lastly, there existed a positive correlation among the scores of the three tests,indicating that the scores of human beings and computers are strongly related.
Abstract
The purpose of this study was to investigate inter- and intra- rater reliability in aninterview and a computerized oral test. It was also examined whether ratercharacteristics influenced on their reliability and biases, and finally the scores of bothtests were compared with those of the Versant test using an automated computer ratingsystem. For the study, the data from 21 Korean university students and 18 Korean ornative speakers of English raters with various characteristics were collected. Some ofthe main findings from the study were as follows. First, rater severity was significantlydifferent in each test, but each rater consistently graded on both tests suggesting lowerinter-rater reliability and higher intra-rater reliability. Secondly, rater severity wasimpacted by the rater characteristics such as mother tongue, gender, age, and major. Lastly, there existed a positive correlation among the scores of the three tests,indicating that the scores of human beings and computers are strongly related.
참고문헌 (28)
- 류경리, "COPI와 PhonePass의 구술 수행평가 타당도 비교 분석", 석사, 영어교육, pp.2006
- 박종임, "Rasch 모형을 활용한 국어교사의 채점 일관성 변화 양상 및 원인 분석 -중학생 서사문 채점을 중심으로-", 우리어문연구, 39(39), pp.301~335 (2011), 우리어문학회
- 신동일, "채점 경향 분석을 위한 Rasch 측정 모형 적용 연구", 외국어 교육, 8(1), pp.249~272 (2001)
- 이영식, "영어작문평가에 대한 채점자 훈련의 원리", 영어교육, 55(2), pp.201~217 (2000)
- 정행, "SOPI를 활용한 영어 구술능력 평가 개발과 결과 분석", 영어교육, 55(2), pp.219~241 (2000)
- 주미진, "컴퓨터 매개 영어 말하기 시험이 수험자 발화에 미치는 영향 조사: 정확성, 유창성, 복잡성을 중심으로", 영어학, 11(3), pp.677~699 (2011), 한국영어학회
- 최인철, "영어 의사소통능력의 모의 구술 면접시험 방식 양상 타당성 검증", 응용언어학, 16(1), pp.215~246 (2000)
- Brown, A., "Test development process : Test‐takers’ reactions to a tape‐mediated test of proficiency in spoken Japanese", Language Testing, 10, pp.277~303 (1993)
- Canale, M, "Theoretical basis of communicative approaches to second language teaching and testing", Applied Linguistics, 1, pp.1~47 (1980)
- "oral proficiency interview" (2002)[URL]
- "Testing/assessment, adult ESL assessment, BEST plus" (2007)[URL]
- Charney, D., "The validity of using holistic scoring to evaluate writing : A critical overview", Research on the Teaching of English, 18(1), pp.65~81 (1984)
- Clark, J. L. D., "Validation of a tape-mediated ACTFL", ILR scale-based test of Chinese speaking proficiency, 1988, pp.5, 187~198
- "Educational Testing Service" (2005)[URL]
- "English Speaking Proficiency Testing Academy" (2002)[URL]
- Fulcher, G., "Testing second language speaking" (2003), Pearson Longman
- 주미진, "The attitudes of students’ and teachers’ toward a Computerized Oral Test (COT) and a Face-To-Face Interview (FTFI) in a Korean u", 언어과학, 14(2), pp.171~193 (2007), 한국언어과학회
- Kenyon, D. M, "Comparing examinee attitudes toward computerassisted and other proficiency assessments", Language Learning and Technology, 5, pp.60~83 (2001)
- Linacre, J. M., "Many Facet Rasch measurement" (1989), MESA Press
- McNamara, T. F., "Measuring second language performance" (1996), Longman
- O'Loughlin, K., "The equivalence of direct and semi-direct speaking tests (studies in language testing 13)" (2001), Cambridge University Press
- Shohamy, E., "The validity of direct versus semi direct oral tests", Language Testing, 7, pp.99~123 (1994)
- Shohamy, E., "The effect of ratings background and training on the reliability of direct writing tests", Modern Language Journal, 76, pp.27~33 (1992)
- Stansfield, C. W., "Current developments in language testing", pp.199~209 (1991), Regional Language Center
- Stansfield, C. W., "The development and validation of the Portuguese speaking test", Hispania, 73, pp.641~651 (1990)
- Weigle, S. C., "Using FACETS to model rater training effects", Paper presented at the 16th Annual Language Testing Research Colloquium (1994)
- Weir, C. J., "Communicative language testing" (1990), Prentice Hall
- Wolcott, W., "An overview of writing assessment: Theory, research, and practice" (1988), National Council of Teachers of English