Examining the Sources of Variability in Writing Test Scores
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
In the present study, in order to better understand the complex nature of writing performance assessment, factors affecting writing test scores were investigated in a systematic analysis in which the effects of these factors could be disentangled. Specifically, the primary purpose of this study was to investigate the extent to which the students’ performance on a writing test was affected by their underlying writing ability as opposed to the test method factors such as specific tasks and raters used to elicit and score test performance. For the purpose of this study, generalizability theory analysis (Brennan, 2001), which enables us to estimate the relative effects of various factors affecting test scores, was used. The results showed that the students’ scores on the writing test were significantly affected by the method effects associated with the writing tasks used. Given the instability of students’ performances across tasks observed in the present study, the way forward seems to recognize that language testers need to take into account both a construct-based and a task-based approach to test design and score interpretation.
Abstract
In the present study, in order to better understand the complex nature of writing performance assessment, factors affecting writing test scores were investigated in a systematic analysis in which the effects of these factors could be disentangled. Specifically, the primary purpose of this study was to investigate the extent to which the students’ performance on a writing test was affected by their underlying writing ability as opposed to the test method factors such as specific tasks and raters used to elicit and score test performance. For the purpose of this study, generalizability theory analysis (Brennan, 2001), which enables us to estimate the relative effects of various factors affecting test scores, was used. The results showed that the students’ scores on the writing test were significantly affected by the method effects associated with the writing tasks used. Given the instability of students’ performances across tasks observed in the present study, the way forward seems to recognize that language testers need to take into account both a construct-based and a task-based approach to test design and score interpretation.
참고문헌 (39)
- Bachman,L.F, "Some reflections on task-based language performance assessment", Language Testing, 19, pp.453~476 (2002)
- Bachman,L.F, "Investigating variability in tasks and rater judgments in a performance test of foreign language speaking", Language Testing, 12, pp.238~257 (1995)
- Bachman,L.F, "Language testing in practice" (1996), Oxford University Press
- Bonk,W.J, "A many-facet Rasch analysis of the second language group oral discussion task", Language Testing, 20, pp.89~110 (2003)
- Brennan,R.L, "Elements of generalizability theory" (1992), ACT
- Brennan,R.L, "Generalizability theory" (2001), Springer-Verlang
- Bridgeman,B, "Survey of academic writing tasks", Written Communication, 1, pp.247~280 (1983)
- Bridgeman,B, "Choice among essay topics:Impact on performance and validity", Journal of Educational Measurement, 34, pp.273~286 (1997)
- Canesco,G, "Writing required in graduate courses in business administration", TESOL Quarterly, 23, pp.305~316 (1989)
- Carlson,J.G, "Relationship of admission test scores to writing performance of native and nonnative speakers of English(TOEFL Research Report No.19)" (1985), Educational Testing Service
- 조윤경, "The Effectiveness of Peer Feedback in Korean EFL Writing Classrooms", 영어교육연구, 17(4), pp.33~59 (2005), 팬코리아영어교육학회
- Cumming,A, "Writing expertise and second language proficiency", Language Learning, 39, pp.81~141 (1989)
- Engelhard,G, "The influence of mode of discourse,experiential demand,and gender on the quality of student writing", Research in the Teaching of English, 26, pp.315~336 (1992)
- Horowitz,D, "Student beliefs about language learning", ~ (1987)
- Kondo-Brown,K, "A FACETS analysis of rater bias in measuring Japanese second language writing performance", Language Testing, 19, pp.3~31 (2002)
- 이호, "영작문 채점 기준표 개발 과정에의 학습자 참여에 대한 연구", 영어교육연구, 19(4), pp.279~11 (2007), 팬코리아영어교육학회
- Leki,I, "Students’ perception of EAP writing instruction and writing needs across the disciplines", TESOL Quarterly, 28, pp.81~101 (1994)
- Lumley,T, "Assessment criteria in a large-scale writing test:What do they really mean to the raters?", Language Testing, 19, pp.246~276 (2002)
- Lynch,B.K, "Using G-theory and many-facet Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants", Language Testing, 15, pp.158~180 (1998)
- McNamara,T.F, "Measuring second language performance" (1996), Longman
- Messick,S, "Validity.In Educational Measurement", pp.13~103 (1989), American Council on Education, Macmillan Publishing Company
- Messick,S, "The interplay of evidence and consequences in the validation of performance assessments", Educational Researcher, 23, pp.13~23 (1994)
- Mislevy,R.J, "Design and analysis in task-based language assessment", Language Testing, 19, pp.477~496 (2002)
- Norris,J.M, "Examinee abilities and task difficulty in task-based second language performance assessment", Language Testing, 19, pp.395~418 (2002)
- 오준일, "영어 말하기 능력 절대 등급 설정에 관한 연구", 영어교육연구, 18(1), pp.141~162 (2006), 팬코리아영어교육학회
- O’Loughlin,K, "Do English and ESL teachers rate essays differently?", Melbourne Papers in Language Testing, 1, pp.28~49 (1992)
- "An investigation of an ESL placement test of writing using Many-facet Rasch measurement.Teachers College,Columbia University Working Papers in TESOL & Applied Linguistics,4(1),Retrieved December 26,2005"[URL]
- Powers,D.E, "Test takers’ judgments of essay prompts:Perceptions and performance", Educational Assessment, 6, pp.3~22 (1999)
- Purves,A, "Reflection on research and assessment in written composition", Research in the Teaching of English, 26, pp.108~122 (1992)
- Quellmalz,E.S, "Effects of discourse and response model on the measurement of writing competence", Journal of Educational Measurement, 19, pp.241~258 (1982)
- Shavelson,R.J, "Generalizability theory:A primer" (1991), Sage
- Shohamy,E, "The effects of raters’ background and training on the reliability of direct writing tests", Modern Language Journal, 76, pp.513~521 (1992)
- Spaan,M, "The effects of prompt on essay examination.In A new decade of language testing research", pp.98~122 (1993), TESOL
- Tyndall,B, "Validation of a new holistic rating scale using Rasch multi-faceted analysis.In Validation in language testing", pp.15~38 (1996), Multilingual Matters
- Vaughan,C, "Holistic assessment:What goes on in the rater’s mind? In Assessing second language writing in academic contexts", pp.111~126 (1992), Ablex
- Webb,N.M, "Multivariate generalizability theory.In New directions in testing and measurement: Generalizability theory", pp.67~82 (1983), Jossey-Bass
- Weigle,S.C, "Effects of training on raters of ESL compositions", Language Testing, 11, pp.197~223 (1994)
- Weigle,S.C, "Using FACETS to model rater training effects", Language Testing, 15, pp.263~287 (1998)
- Weigle,S.C, "Integrating reading and writing in a competency test for non-native speakers of English", Assessing Writing, 9, pp.27~55 (2004)