영작문 평가를 위한 채점자 훈련의 방향
How should English teachers as raters be trained for writing assessment?
1
조회
0
다운로드
9
인용
0
참고문헌
키워드
초록
This paper aimed to find what English teachers at secondary schools in Korea did during their rating assessment and to suggest what effective rater training session/programme for them should be like. Three English teachers at various secondary schools participated as raters for this study, and they were asked to rate six pieces of writing samples written by secondary school students, using FCE rating scale for writing assessment. They were also supposed to do think-aloud while rating so as to reveal their rating behaviour, and then their protocols recorded on cassette tapes were transcribed. The results showed that; the raters did not prevent themselves from relying on their subjective criteria; they failed to stick to the scale throughout their scoring; they were likely to rate a sample, comparing with preceeding samples; they often had difficulties in understanding some descriptors within the scale; the very first impression which a sample made on a rater often affected on his/her rating; they sometimes showed halo effect in their rating. From these findings it can be suggested that rater training session or programme for English teachers should be devised, reflecting their rating behaviour and help them improve test validity, and further intra- and inter-rater reliability.
Abstract
This paper aimed to find what English teachers at secondary schools in Korea did during their rating assessment and to suggest what effective rater training session/programme for them should be like. Three English teachers at various secondary schools participated as raters for this study, and they were asked to rate six pieces of writing samples written by secondary school students, using FCE rating scale for writing assessment. They were also supposed to do think-aloud while rating so as to reveal their rating behaviour, and then their protocols recorded on cassette tapes were transcribed. The results showed that; the raters did not prevent themselves from relying on their subjective criteria; they failed to stick to the scale throughout their scoring; they were likely to rate a sample, comparing with preceeding samples; they often had difficulties in understanding some descriptors within the scale; the very first impression which a sample made on a rater often affected on his/her rating; they sometimes showed halo effect in their rating. From these findings it can be suggested that rater training session or programme for English teachers should be devised, reflecting their rating behaviour and help them improve test validity, and further intra- and inter-rater reliability.
참고문헌 (29)
- 김형엽, "영어작문 교육에서 공동협력자로서의 교사의 역할: 교사-학생면담을 중심으로", 현대영어교육, 7(2), pp.96~129 (2006)
- 박경자, "응용언어학 사전" (2001), 경진문화사
- 박상옥, "영어교육에서 베껴 쓰기와 요약하여 쓰기의 효과 비교: 사례연구를 중심으로", 현대영어교육, 10(2), pp.60~86 (2009), 현대영어교육학회
- 이영식, "영어작문 평가에 대한 채점자 훈련의 원리", 영어교육, 55(2), pp.35~56 (2000)
- 이지연, "영어 교사들의 주관적 평가 방법에 따른 영작문 평가 시 평가항목의 타당도", 응용언어학, 23(1), pp.147~171 (2007), 한국응용언어학회
- 최연희, "채점자 훈련이 영어 작문 채점에 미치는 효과에 대한 연구: FACETS 분석을 통한 신뢰도 변화 분석", 응용언어학, 18(1), pp.13~292 (2002), 한국응용언어학회
- Alderson, J. C., "Bands and scores In: Language testing in the 1990s: The communicative legacy", pp.71~86 (1991), Macmillan
- Brown, J. D., "Do English and ESL faculties rate writing samples differently?", TESOL Quarterly, 25(4), pp.587~603 (1991)
- Brown, A., "The effect of rater variables in the development of an occupation-specific language performance test", Language Testing, 12(1), pp.1~15 (1995)
- Carrell, P. L., "The effect of writers' personalities and raters' personalities on the holistic evaluation of writing", Assessing Writing, 2(2), pp.153~190 (1995)
- 조수경, "Patterns of Sentence Connectors and the Effect of Instruction on ESL Learners' Writing", 현대영어교육, 10(2), pp.1~22 (2009), 현대영어교육학회
- Davies, A., "Validity and validation in language testing In: Handbook of research in second language teaching and learning", pp.795~813 (2005), Lawrence Erlbaum
- Freedman, S. W., "Influences on evaluators of expository essays: Beyond the test", Research in the teaching of English, 15, pp.244~255 (1981)
- Glendinning, E., "Examining the intangible process: Lotus ScreenCam as an aid to investigating student writing", Edinburgh Working Papers in Applied Linguistics, 11, pp.42~58 (2001)
- Gowen, S., "Writing, rating and personality type", Paper presented at the ninth annual University System of Georgia Developmental Studies Conference (1984)
- Green, A., "Verbal protocol analysis in language testing research: A handbook (vol. 5)" (1998), Cambridge University Press
- Hamp-Lyons, L., "Second language writing: assessment issues In: Second language writing: research insights for the classroom", pp.69~87 (1990), Cambridge University Press
- Hill, K., "Who should be the judge? The use of non-native speakers as raters on a test of English as an international language In: Current developments and alternatives in language assessment-Proceedings of LTRC 96", pp.275~290 (1997), University of Jyvaskyla and University of Tampere.
- Hill, K., "Analytic rating scales: how diagnostic are they?", Melbourne Papers in Language Testing, 3(1), pp.50~65 (1994)
- Jensen, G. H., "Personality and the teaching of composition" (1989), Ablex
- Kondo-Brown, K., "A FACETS analysis of rater bias in measuring Japanese second language writing performance", Language Testing, 19(1), pp.3~31 (2002)
- Lumley, T., "The judgements of language-trained raters and doctors in a test of English for health professionals", Melbourne Papers in Language Testing, 4(1), pp.74~98 (1995)
- O'Loughlin, K., "Do English and ESL teachers rate essays differently?", Melbourne Papers in Language Testing, 1(2), pp.19~44 (1992)
- Upshur, J. A., "Constructing rating scales for second language tests", ELT Journal, 49(1), pp.3~12 (1995)
- Warschauer, M., "Automated writing evaluation: defining the classroom research agenda", Language Teaching Research, 10(2), pp.157~180 (2006)
- Weigle, S. C., "Effects of training on raters of ESL compositions", Language Testing, 11, pp.197~223 (1994)
- Weigle, S. C., "Using FACETS to model rater training effects", Language Testing, 15(2), pp.263~287 (1998)
- Weigle, S. C., "Assessing writing" (2002), Cambridge University Press
- Yi, Jyi-yeon, "Construction of a rating scale for writing assessment in an EFL context (Unpublished)", Doctor, pp.2006