Applicability of an EBB Scale to English Speaking Performance Assessment in the Korean Secondary School
0
조회
0
다운로드
5
인용
0
참고문헌
키워드
초록
This study investigates not only some of the structural issues of an EBB rating scale but also rater reliability in terms of its stability in the classroom context. A group of three teachers developed two EBB scales: a four-level EBB for linguistic accuracy and a six-level EBB for communicative effectiveness. The findings on structural traits of an EBB scale are: i) the binary question format of an EBB scale does not prevent confusion in perceiving some subjective linguistic factors; ii) the top questions exert too influential power on subsequent questions or sub-groups; iii) an EBB scale is so inflexible that it cannot reflect linguistic traits that do not appear in a particular level; iv) the limited number of questions cannot cover some unique linguistic traits of a few examinees. However, a FACETS analysis shows that an EBB scale is effective in ensuring high rater reliability. Raters show similar rater severity, enough intra-rater reliability without any misfit or overfit, and high ICC inter-rater reliability with .83 for linguistic accuracy and with .89 for communicative effectiveness.
Abstract
This study investigates not only some of the structural issues of an EBB rating scale but also rater reliability in terms of its stability in the classroom context. A group of three teachers developed two EBB scales: a four-level EBB for linguistic accuracy and a six-level EBB for communicative effectiveness. The findings on structural traits of an EBB scale are: i) the binary question format of an EBB scale does not prevent confusion in perceiving some subjective linguistic factors; ii) the top questions exert too influential power on subsequent questions or sub-groups; iii) an EBB scale is so inflexible that it cannot reflect linguistic traits that do not appear in a particular level; iv) the limited number of questions cannot cover some unique linguistic traits of a few examinees. However, a FACETS analysis shows that an EBB scale is effective in ensuring high rater reliability. Raters show similar rater severity, enough intra-rater reliability without any misfit or overfit, and high ICC inter-rater reliability with .83 for linguistic accuracy and with .89 for communicative effectiveness.
참고문헌 (35)
- Bachman,L.F, "Language testing in practice:Designing and developing useful language tests" (1996), Oxford University Press
- Bachman,L.F, "The evaluation of communicative language proficiency:A critique of the ACTFL oral interview", The Modern Language Journal, 70(4), pp.380~390 (1986)
- Brindley,G, "Describing language development? Rating scales and SLA.In Interfaces between second language acquisition and language testing research", pp.112~140 (1998), Cambridge University Press
- Brown,J.D, "Criterion-referenced language testing" (2002), Cambridge University Press
- Chalhoub-Deville,M, "Assessment measurement for specific contexts of language use.Revised version of a paper presented at the 18th American Association for Applied Linguistics", ERIC Document No. ED 401721, ~ (1996)
- Chapelle,C, "Construct definition and validity inquiry in SLA research.In Interfaces between second language acquisition and language testing research", pp.32~70 (1998), Cambridge University Press
- Douglas,D, "Testing methods in context-based second language research.In Interfaces between SLA and language testing research", pp.141~155 (1998), Cambridge University Press
- Ellis,R, "Task-based research and language pedagogy", Language Teaching Research, 4(3), pp.193~220 (2000)
- Fagot,R.F, "Reliability of ratings for multiple judges:Intraclass correlation and metric scales", Applied Psychological Measurement, 15(1), pp.1~11 (1991)
- Fulcher,G, "Invalidating validity claims for the ACTFL oral rating scale", System, 24(2), pp.163~172 (1996)
- Fulcher,G, "Testing second language speaking" (2003), Pearson Education
- Fulcher,G, "Evaluating quality in second language performance tests", English Language Assessment, 1, pp.3~19 (2007)
- Fulcher,G, "Language testing and assessment:An advanced resource book" (2007), Routledge
- "Reliability analysis.Retrieved November 22,2008"[URL]
- Harris,T, "Exploring English 2" (1995), Longman
- Hudson,T, "Trends in assessment scales and criterion-referenced language assessment", Annual Review of Applied Linguistics, 25, pp.205~227 (2005)
- 황성삼, "EBB를 이용한 중등학교 말하기 평가척도 개발", 영어교육, 63(2), pp.307~331 (2008), 한국영어교육학회
- Iwashita,N, "Can we predict task difficulty in an oral proficiency test? Exploring the potential of an information processing approach to task design", Language Learning, 21, pp.401~436 (2001)
- 전병만, "영어과 교사의 학생 평가 전문성 신장을 위한 연구", 영어교육, 60(2), pp.3~27 (2005), 한국영어교육학회
- Kim,D.K, "A proposal for performance assessment criteria with focus on speaking and writing abilities", English Teaching, 54(2), pp.175~200 (1999)
- Lantolf,J, "Oral proficiency testing:A critical analysis", Modern Language Journal, 69, pp.337~45 (1985)
- Lazaraton,A, "A qualitative approach to the validation of oral language tests" (2002), Cambridge University Press
- Lee,C.S, "High School ENGLISH" (2001), Neungyul Gyoyuk
- "A user’s guide to Facets:Rasch measurement computer program.Retrieved July 25,2008"[URL]
- McNamara,T.F, "Measuring second language performance" (1996), Longman
- MEHRD, "Godeunghakgyo yeongeogwa gyoyukgwajeong unyeong jaryo(Curriculum for English language in high school)", Ministry of Education & Human Resources Development, Curriculum material, No. 205. Sechangmunhwasa, ~ (2003)
- North,B, "Scaling descriptors for language proficiency scales", Language Testing, 15(2), pp.217~263 (1998)
- 오준일, "영어 말하기 능력 절대 등급 설정에 관한 연구", 영어교육연구, 18(1), pp.141~162 (2006), 팬코리아영어교육학회
- Shin,D.I, "Revising an ACTFL-SOPI formatted speaking test in Korea:Problems and suggestion", English Teaching, 56(2), pp.309~331 (2001)
- 신동일, "Rasch 모형을 이용한 고등학교 영어과 말하기 및 쓰기 능력 등급 기술표 개발", 영어교육, 57(4), pp.469~499 (2002), 한국영어교육학회
- 신동일, "SLA 선행 연구에 근거한 말하기 평가척도 제안: 문법 영역 중심으로", 영어교육, 57(2), pp.423~450 (2002), 한국영어교육학회
- Skehan,P, "A framework for the implementation of task-based instruction", Applied Linguistics, 17, pp.38~62 (1996)
- Song,D.Y, "An analysis of assessment domains and methods used in English performance assessment in high schools", The Journal of English Education, 34, pp.95~124 (2007)
- Upshur,J, "Constructing rating scales for second language tests", ELT Journal, 49, pp.3~12 (1995)
- Upshur,J, "Systematic effects in the rating of second-language speaking ability:Test method and learner discourse", Language Testing, 16(1), pp.82~111 (1999)