Investigating the Reliability of an Analytic Rubric to Score TOEFL Independent Writing
1
조회
0
다운로드
2
인용
0
참고문헌
키워드
초록
The current study aims to design The current study aims to design an analytic writing scale with reference to the TOEFL iBT® Scoring rubric from the Educational Testing Service (ETS) and to examine the reliability of the newly designed analytic rubric using English argumentative writing collected from 26 Korean undergraduate and graduate students. There are three main findings in this study: First, intra-rater reliability was examined among three raters showing a high correlation between the holistic and the analytic scorings. Second, inter-rater reliability was explored in terms of correlation coefficients, agreement rates (adjacent and perfect), Cohen’s Kappa coefficients, and Fleiss Kappa coefficients. In addition, the intraclass correlation coefficient (ICC) revealed a high correlation between the score on a new analytic rubric and the original holistic TOEFL score. Among the four rubric dimensions, ‘unity-progression-coherence’ and ‘language use’ showed relatively lower correlations than the other two dimensions (topic and task, organization and development). Also, the dimension that showed the highest correlation with the holistic ETS score is ‘organization and development.’ Moreover, ‘unity-progression-coherence’ contributed notably to the analytic assessment. A survey on raters’ perceptions of rubrics and scoring, and interviews suggested that they had challenges scoring based on the descriptors of two rubrics (analytic and holistic). This study has pedagogical implications for scoring writing, designing rubrics, and rater training.
Abstract
The current study aims to design The current study aims to design an analytic writing scale with reference to the TOEFL iBT® Scoring rubric from the Educational Testing Service (ETS) and to examine the reliability of the newly designed analytic rubric using English argumentative writing collected from 26 Korean undergraduate and graduate students. There are three main findings in this study: First, intra-rater reliability was examined among three raters showing a high correlation between the holistic and the analytic scorings. Second, inter-rater reliability was explored in terms of correlation coefficients, agreement rates (adjacent and perfect), Cohen’s Kappa coefficients, and Fleiss Kappa coefficients. In addition, the intraclass correlation coefficient (ICC) revealed a high correlation between the score on a new analytic rubric and the original holistic TOEFL score. Among the four rubric dimensions, ‘unity-progression-coherence’ and ‘language use’ showed relatively lower correlations than the other two dimensions (topic and task, organization and development). Also, the dimension that showed the highest correlation with the holistic ETS score is ‘organization and development.’ Moreover, ‘unity-progression-coherence’ contributed notably to the analytic assessment. A survey on raters’ perceptions of rubrics and scoring, and interviews suggested that they had challenges scoring based on the descriptors of two rubrics (analytic and holistic). This study has pedagogical implications for scoring writing, designing rubrics, and rater training.
참고문헌 (49)
- Bacha, N., "Writing evaluation: What can analytic versus holistic essay scoring tell us?", System, 29, pp.371~383 (2001)
- Bachman, L. F., "Language testing in practice: Designing and developing useful langauge test" (1996), Oxford University Press
- Barkaoui, K., "Raring scale impact on EFL essay marking : A mixed-method study", Assessing Writing, 12, pp.86~107 (2002)
- Bauer, B. A., "A study of the reliabilities and the cost-efficiencies of three methods of assessment for writing ability", pp.1981
- Becker, A., "Examining rubrics used to measure writing performance in US intensive English programs", The CATESOL Journal, 22(1), pp.113~130 (2011)
- Brown, H. D., "Language Assessment: Principles and classroom practice" (2010), Pearspn Longman
- 변정희, "Validating Writing Scale for High School English with Common European Framework of Reference", 영어교육연구, 24(2), pp.27~48 (2012), 팬코리아영어교육학회[DOI]
- 정선주, "Investigating Linguistic Features of Cohesion in Korean EFL College Writing", 영어영문학21, 33(3), pp.211~233 (2020), 21세기영어영문학회[DOI]
- Coombe, C., "The Cambridge guide to second language assessment" (2012), Cambridge University Press
- Davies, A., "Handbook of research in second language teaching and learning", pp.795~813 (2005), Lawrence Erlbaum
- Factors in judgments of writing ability, "Diederich, P. B.", Educational Testing Service, ~, pp.Research Bulletin RB-61-15 (1961)
- Douglas, D, "Theoretical underpinnings of the Test of Spoken. English Revision Project", TOEFL Monograph Series, (9) (1997), ETS
- East, M., "Evaluating the reliability of a detailed analytic scoring rubric for foreign language writing", Assessing Writing, 14, pp.88~115 (2009)
- Feldt, L. S., "Educational measurement", pp.105~146 (1989), The American Council on Education and the National Council on Measurement in Education
- Fulcher, G., "Encyclopedia of language education, Vol. 7: Language testing and assessment", pp.75~78 (1997), Kluwer Academic Publishers
- Ghalib, T. K., "Holistic versus Analytic Evaluation of EFL Writing: A Case Study", English Language Teaching, 8(7), pp.225~236 (2015), Heinle & Heinle
- Hamp-Lyons, L., "Assessing Second Language Writing in Academic Contexts", pp.279~291 (1991), Ablex Publishing Corporation
- Hamp-Lyons, L., "Research on the rating process : Rating nonnative writing : The trouble with holistic scoring", TESOL Quarterly, 29, pp.759~762 (1995)
- Hamp-Lyons, L, "Editorial worrying about rating", Assessing Writing, 12, pp.1~9 (2007)
- Humphry, S., "Raters’ perceptions of assessment criteria relevance", Assessing Writing, 41, pp.1~13 (2019)
- Jescovitch, L. N., "Deconstruction of Holistic Rubrics into Analytic Rubrics for Large-Scale Assessments of Students’Reasoning of Complex Science Concepts", Practical Assessment, Research, and Evaluation, 24(7), pp.1~13 (2019)
- Johnson, D. M., "Research on the rating process : Rating nonnative writing : The trouble with holistic scoring", Tesol Quarterly, 29(4), pp.759~762 (1995)
- Kane, M. T., "Testing in the profession: Credentialing policies and practice", pp.20~41 (2017), Routledge
- Klein, S. P., "Analytic versus holistic scoring of science performance tasks", Applied Measurement in Education, 1, pp.121~137 (1998)
- Knoch, U., "Diagnostic assessment of writing : A comparison of two rating scales", Language Testing, 26, pp.275~304 (2009)
- Kuo, S., "Which rubric is more suitable for NSS liberal studies? Analytic or holistic?", Educational Research Journal, 22(2), pp.179~199 (2007)
- 이호, "척도의 활용이 영작문 평가에 미치는 영향—예비영어교사의 평가전문성 신장의 관점에서", 영어영문학21, 34(4), pp.363~388 (2021), 21세기영어영문학회[DOI]
- Lea, M., "Student writing in higher education : an academic literacies approach", Studies in Higher Education, 23(2), pp.157~172 (1998)
- Markham, L. R., "Influences of handwriting quality on teacher evaluation of written work", American Educational Research Journal, 13(4), pp.277~283 (1976)
- Messick, S., "Educational measurement", pp.13~103 (1989), Macmillan
- Yvette Denise Murdoch, "Direct Instruction and Use of Online English Writing Software on EMI Class-Takers’ Self-Efficacy", International Journal of Contents, 15(4), pp.97~106 (2019), 한국콘텐츠학회[DOI]
- Nakamura, Y, "A comparison of holistic and analytic scoring methods in the assessment of writing. The Interface Between Interlanguage", Pragmatics and Assessment: Proceedings of the 3rd Annual JALT Pan-SIG Conference (2004), Tokyo Keizai University
- Nakkaew, N., "An initial development of an analytic rubric for assessing critical thinking in English argumentative essays of EFL college students", Reflections, 26(2), pp.51~74 (2019)
- Ono, M., "Holistic and analytic assessments of the TOEFL iBT® integrated writing task", JLTA(Japan Language Testing Association)Journal, 22, pp.65~88 (2019)
- Park, J. G., "The study on the rater reliability of three scoring methods in assessing argumentative essays : Holistic, analytic, and multiple-trait scoring methods", Studies on Foreign Language Education, 9, pp.63~84 (2006)
- Shabani, E. A., "Examining consistency among different rubrics for assessing writing", Language Testing in Asia, 10(12), pp.1~25 (2020)
- Shaw, S. D., "Examining writing : research and practice in assessing second language writing" (2007), Cambridge University Press
- Shi, L., "Native-and nonnative-speaking EFL teachers’ evaluation of Chinese students’ English writing", Language Testing, 18, pp.303~325 (2001)
- Schoonen, R., "Generalizability of writing scores : An application of structural equation modeling", Language Testing, 22, pp.1~30 (2005)
- Turner, C. E., "Rating scales derived from students samples : Effects of the scale maker and the student sample on scale content and students scores", TESOL Quarterly, 36, pp.49~70 (2002)
- Underhill, N., "Testing spoken language : A handbook of oral testing techniques" (1987), Cambridge University Press
- Upshur, J. A., "Constructing rating scales for Second Language Test", ELT Journal, 49, pp.3~12 (1995)
- Vacc, N. N., "Writing evaluation : Examining four teachers’ holistic and analytic scores", The Elementary School Journal, 90, pp.87~95 (1989)
- Vinas, L. F., "Testing the reliablity of two rubrics used in official English certificates for the assessment of writing", Alicante Journal of English Studies, 36, pp.85~109 (2022)
- Weigle, S. C., "Assessing writing" (2002), Cambridge University Press
- White, E., "Holisticism", College Composition and Communication, 35, pp.400~409 (1984)
- Wu, S. M., "Creating a contrastive rhetorical stance : investigating the strategy of problematization in students’ argumentation", RELC Journal, 37(3), pp.329~353 (2006)
- 이지연, "고등학생들의 영작문 평가를 위한 평가 척도 개발의 필요성", Foreign Languages Education, 15(3), pp.261~286 (2008), 한국외국어교육학회
- Zhang, B., "Rater reliability and score discrepancy under holistic and analytic scoring of second language writing", Language Testing in Asia, 5(5), pp.1~9 (2015)