Exploring the Role of Automatically-derived Text Complexity Features in L2 Reading Test Development
0
조회
0
다운로드
2
인용
0
참고문헌
키워드
초록
Although automatic text analysis tools are available, little research has been conducted on the application of such tools in reading assessments. When the ratio of academic vocabulary and transitions are computed automatically and used in test development, the text selection-revision procedure can be fast and transparent by complementing test developers’ expertise. To obtain empirical evidence for the utility of automatic text complexity features, this study attempted to explore the role of automatically-derived text complexity features in an intensive English program (IEP) reading assessment. Based on previous literature and the testing context, a total of 11 text complexity features as lexical, syntactic, and semantic variables were chosen, and their accountability for the IEP reading item difficulty was automatically measured by using three text analysis tools—Lexile, the Compleat Lexical Tutor, and Coh-Metrix. Results showed that seven complexity features significantly correlated with the reading item difficulty. Stepwise multiple regressions showed that a set of four lexical and semantic text complexity features (i.e., word length, total word counts, Lexical Semantic Analysis (LSA), connectives) explained about 45% of the variance in the reading item difficulty. The results and findings of this study are discussed with regard to limitations and implications for both reading assessments and instruction.
Abstract
Although automatic text analysis tools are available, little research has been conducted on the application of such tools in reading assessments. When the ratio of academic vocabulary and transitions are computed automatically and used in test development, the text selection-revision procedure can be fast and transparent by complementing test developers’ expertise. To obtain empirical evidence for the utility of automatic text complexity features, this study attempted to explore the role of automatically-derived text complexity features in an intensive English program (IEP) reading assessment. Based on previous literature and the testing context, a total of 11 text complexity features as lexical, syntactic, and semantic variables were chosen, and their accountability for the IEP reading item difficulty was automatically measured by using three text analysis tools—Lexile, the Compleat Lexical Tutor, and Coh-Metrix. Results showed that seven complexity features significantly correlated with the reading item difficulty. Stepwise multiple regressions showed that a set of four lexical and semantic text complexity features (i.e., word length, total word counts, Lexical Semantic Analysis (LSA), connectives) explained about 45% of the variance in the reading item difficulty. The results and findings of this study are discussed with regard to limitations and implications for both reading assessments and instruction.
참고문헌 (20)
- Alderson, C. C., "Assessing reading" (2000), Cambridge University Press
- Castello, E., "Text complexity and reading comprehension tests" (2008), Peter Lang
- "The Compleat Lexical Tutor"[URL]
- Crossley, S. A., "Assessing text readability using cognitively based indices", TESOL Quarterly, 43(3), pp.475~492 (2008)
- Drum, P. A., "The effects of surface structure variables on performance in reading comprehension tests", Reading Research Quarterly, 16(4), pp.486~514 (1981)
- Embretson, S. E., "Component latent trait models for paragraph comprehension tests", Applied Psychological Measurement, 11(2), pp.175~193 (1987)
- Fisher, D., "Addressing CCSSO anchor standard 10: Text complexity", Language Arts, 91(4), pp.250~250 (2014)
- Fisher, D., "Text complexity: Raising rigor in reading" (2012), International Reading Association
- Freedle, R., "The prediction of TOEFL reading item difficulty: Implications for construct validity", Language Testing, 10(2), pp.133~170 (1993)
- Graesser, A. C., "Coh-Metrix: Analysis of text on cohesion and language", Behavior Research Methods, Instruments, & Computers, 36(2), pp.193~202 (2004)
- Gray, W. S., "What makes a book readable?" (1935), University of Chicago Press
- IBM SPSS Statistics for Windows, Version 22.0, "IBM Corp", IBM Corp, ~ (2013)
- 전문기, "자동화된 언어분석 프로그램을 활용한 중학교 영어교과서의 학년 간 연계성 분석", 현대영어교육, 16(1), pp.195~218 (2015), 현대영어교육학회
- 전문기, "코메트릭스(Coh-Metrix)를 이용한 중학교 1학년 개정 영어 교과서의 코퍼스 언어학적 비교 분석", 영어교육연구, 21(4), pp.265~292 (2009), 팬코리아영어교육학회[DOI]
- 김재은, "고등학교 영어 교과서, EBS 수능 연계 교재, 대학수학능력시험의 코퍼스기반 난이도 비교 분석", 멀티미디어 언어교육, 18(1), pp.59~92 (2015), 한국멀티미디어언어교육학회[DOI]
- Livingston, S. A., "Delta equating" (2013), Educational Testing Service
- McNamara, D. S., "Assessing reading in the 21th century: Aligning and applying advances in the reading and measurement sciences", pp.89~116 (2012), R & L Education
- McNamara, D. S., "Learning from text: Effects of prior knowledge and text coherence", Discourse Processes, 22(3), pp.247~287 (1996)
- "Common Core State Standards for English language arts and literacy in history/social studies, science, and technical subjects: Appendix A: Research supporting key elements of the standards and glossary of key terms" (2010)[URL]
- Tabachnick, B. G., "Using multivariate statistics" (2007), Pearson Education, Inc