Exploring the Linguistic and Textual Factors That Affect the Quality of L2 Writing: The Coh-Metrix Study
1
조회
0
다운로드
3
인용
0
참고문헌
키워드
초록
Efforts have been made to improve the output of automated essay evaluation systems to emulate human ratings in the field of applied linguistics. The current study aimed to explore the textual features that distinguish good from poor second-language (L2) writing using Coh-Metrix version 3.0 and to compare the output of an online text analysis tool to human rating. Sixty essays were collected from tertiary-level on-campus essay writing contests in 2017 - 2018. Three experienced English instructors rated the essays using a rubric from Jacobs et al. (1981). The Coh-Metrix output, including indices of descriptive, text easability, lexical diversity, connectives, latent semantic analysis (LSA), word information, and L2 readability, were evaluated by correlation analyses and independent t-tests. The results showed that higher scorers wrote longer, with a greater number of sentences and paragraphs. They were also likely to use more concrete words, more frequent words, and better cohesion, all of which contributed to readability. The pedagogical implications and limitations of using Coh-Metrix for L2 writing testing and instruction were further discussed.
Abstract
Efforts have been made to improve the output of automated essay evaluation systems to emulate human ratings in the field of applied linguistics. The current study aimed to explore the textual features that distinguish good from poor second-language (L2) writing using Coh-Metrix version 3.0 and to compare the output of an online text analysis tool to human rating. Sixty essays were collected from tertiary-level on-campus essay writing contests in 2017 - 2018. Three experienced English instructors rated the essays using a rubric from Jacobs et al. (1981). The Coh-Metrix output, including indices of descriptive, text easability, lexical diversity, connectives, latent semantic analysis (LSA), word information, and L2 readability, were evaluated by correlation analyses and independent t-tests. The results showed that higher scorers wrote longer, with a greater number of sentences and paragraphs. They were also likely to use more concrete words, more frequent words, and better cohesion, all of which contributed to readability. The pedagogical implications and limitations of using Coh-Metrix for L2 writing testing and instruction were further discussed.
참고문헌 (62)
- 안수진, "Discriminating L2 Writing Proficiency through the Use of Computational Tool Coh-Metrix", 언어과학연구, (92), pp.209~232 (2020), 언어과학회
- Aryadoust, V., "Predicting EFL writing ability from levels of mental representation measured by Coh-Metrix : A structural equation modeling study", Assessing Writing, 24, pp.35~58 (2015)
- Blake, R. J., "The use of technology for second language distance learning", The Modern Language Journal, 93, pp.822~835 (2009)
- Blass, L., "Mosaic one: Writing paragraph review and essay development" (2007), McGraw-Hill
- Crossley, S. A., "The development and use of cohesive devices in L2 writing and their relations to judgments of essay quality", Journal of Second Language Writing, 32, pp.1~16 (2016)
- Crossley, S. A., "Computational assessment of lexical differences in L1 and L2 writing", Journal of Second Language Writing, 18(2), pp.119~135 (2009)
- Crossley, S. A., "Understanding expert ratings of essay quality: Coh-Metrix analyses of first and second language writing", International Journal of Continuing Engineering Education and Life-Long Learning, 21(2, pp.3) (2011), 170~191
- Crossley, S. A., "Moving beyond classic readability formulas: New methods and new models", Journal of Research in Reading, 42(3-4), pp.541~561 (2019)
- Crossley, S. A., "What's so simple about simplified texts? A computational and psycholinguistic investigation of text comprehension and text processing", Reading in a Foreign Language, 26(1), pp.92~113 (2014)
- Cumming, A., "Decision making while rating ESL", EFL writing tasks : A descriptive framework, 2002, pp.86(1), 67~96
- Dale, E., "A formula for predicting readability : Instructions", Educational Research Bulletin, 27(1), pp.37~54 (1948)
- Evangelopoulos, N., "Latent semantic analysis : Five methodological recommendations", European Journal of Information Systems, 21(1), pp.70~86 (2012)
- Ferrari, M., "What makes a good writer? Differences in good and poor writers' self-regulation of writing", Instructional Science, 26, pp.473~488 (1998)
- Ferris, D., "Error feedback in L2 writing classes", Journal of Second Language Writing, 10(3), pp.161~184 (2001)
- Ferster, B., "Automated formative assessment as a tool to scaffold student documentary writing", Journal of Interactive Learning Research, 23(1), pp.81~99 (2012)
- Goh, T. T., "Microfeatures influencing writing quality : The case of Chinese students’ SAT essay", Computer Assisted Language Learning, 33(4), pp.455~481 (2020)
- 구지혜, "Transfer of L1 writing proficiency among EFL students: Testing the threshold hypothesis with Korean 9^th grade EFL learners", 언어학연구, 17(2), pp.135~156 (2012), 한국언어연구학회[DOI]
- Graesser, A. C., "Coh-Metrix measures text characteristics at multiple levels of language and discourse", The Elementary School Journal, 115(2), pp.210~229 (2014)
- Graesser, A. C., "Coh-Metrix: Analysis of text on cohesion and language", Behavior Research Methods, Instruments, & Computers, 36(2), pp.193~202 (2004)
- "Exploring the affordances of Numbas for mathematical learning: A case study" (2020)[URL]
- Halliday, M. A. K., "Cohesion in English" (2014), Routledge
- Hongwei, W., "A computational analysis of textual features and L2 writing proficiency", International Journal of Academic Research in Progressive Education and Development, 2(4), pp.170~185 (2013)
- Housen, A., "Complexity, accuracy, and fluency in second language acquisition", Applied Linguistics, 30(4), pp.461~473 (2009)
- Intaraprawat, P., "The use of metadiscourse in good and poor ESL essays", Journal of Second Language Writing, 4(3), pp.253~272 (1995)
- Jacobs, H. L., "Testing ESL composition: A practical approach" (1981), Newbury House
- 정연주, "Predicting Second Language Writing Proficiency in Learner Texts Using Computational Tools", Journal of Asia TEFL, 16(1), pp.37~52 (2019), 아시아테플
- Kobrin, J. L., "Contextual factors associated with the validity of SAT scores and high school GPA for predicting first-year college grades", Educational Assessment, 16(4), pp.207~226 (2011)
- 공은정, "L2 Proficiency Effect on the Acoustic Cue-Weighting Pattern by Korean L2 Learners of English:Production and Perception of English Stops", 말소리와 음성과학, 5(4), pp.81~90 (2013), 한국음성학회[DOI]
- Larsen-Freeman, D., "The emergence of complexity, fluency, and accuracy in the oral and written production of five Chinese learners of English", Applied Linguistics, 27(4), pp.590~619 (2006)
- 이은희, "An Analysis of Korean EFL University Learners’ Use of Cohesion in Writing with Coh-Metrix", 외국학연구, (27), pp.195~216 (2014), 외국학연구소[DOI]
- Lee, J-W., "The relative contribution of L2 language proficiency and L1 reading ability to L2 reading performance : A test of the threshold hypothesis in an EFL context", TESOL Quarterly, 31(4), pp.713~739 (1997)
- Lentz, L., "The evaluation of text quality : Expert-focused and reader-focused methods compared", IEEE Transactions on Professional Communication, 40(3), pp.224~234 (1997)
- 임효정, "Exploring the Development of Second Language (L2) Vocabulary in Speaking", 외국어교육연구, 37(1), pp.145~163 (2023), 외국어교육연구소[DOI]
- Lim, H., "Automatization in second language sentence processing : A partial, conceptual replication of Hulstijn, Van Gelderen, and Schoonen’s 2009 study", Applied Psycholinguistics, 36(5), pp.1247~1282 (2014)
- Lim, H., "Review of criterion for English language learning", Language Learning & Technology, 16(2), pp.38~45 (2012)
- Lu, X., "Automatic analysis of syntactic complexity in second language writing", International Journal of Corpus Linguistics, 15(4), pp.474~496 (2010)
- MacArthur, C. A., "Which linguistic features predict quality of argumentative writing for college basic writers, and how do those features change with instruction?", Reading and Writing, 32(6), pp.1553~1574 (2018)
- Machicao, J. C, "Information technology and systems:Advances in intelligent systems and computing", pp.835~844 (2019), Springer
- Malmcrona, H, "Variation in assessment: A Coh-Metrix analysis of evaluation of written English in the Swedish upper secondary school", master, pp.2020
- McCarthy, P. M., "MTLD, vocd-D, and HD-D : A validation study of sophisticated approaches to lexical diversity assessment", Behavior Research Methods, 42(2), pp.381~392 (2010)
- McNamara, D. S., "Applied natural language processing and content analysis: Identification, investigation and resolution", pp.188~205 (2012), IGI Global
- McNamara, D. S., "Coh-Metrix : Capturing linguistic features of Cohesion", Discourse Processes, 47(4), pp.292~330 (2010)
- Mody, M. E., "Brain, behavior, and learning in language and reading disorders" (2008), Guilford Press
- 박묘영, "Exploring the Raters’ Bias on an EFL Writing Assessment Using Multi-faceted Rasch Measurement", Studies in English Education, 17(2), pp.175~202 (2012), 글로벌영어교육학회
- Petchprasert, A., "Utilizing an automated tool analysis to evaluate EFL students’ writing performances", Asian-Pacific Journal of Second and Foreign Language Education, 6(1), pp.1~16 (2021)
- Peters, E, "The Routledge handbook of vocabulary studies", pp.125~142 (2019), Routledge
- Pitler, E., "Revisiting readability: A unified framework for predicting text quality", Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pp.86~195 (2008), Association for Computational Linguistics
- Plag, I., "Introduction to English Linguistics" (2015), Walter de Gruyter GmbH & Co KG
- Plakans, L., "An assessment perspective on argumentation in writing", Journal of Second Language Writing, 36, pp.85~86 (2017)
- Rao, P. S., "The significance of writing skills in ELL environment", Academician : An International Multidisciplinary Research Journal, 9(3), pp.5~17 (2019)
- Sawaki, Y., "Understanding learner strengths and weaknesses : Assessing performance on an integrated writing task", Language Assessment Quarterly, 10(1), pp.73~95 (2013)
- Schaefer, E., "Rater bias patterns in an EFL writing assessment", Language Testing, 25(4), pp.465~493 (2008)
- Schmitt, N., "Instructed second language vocabulary learning", Language Teaching Research, 12(3), pp.329~363 (2008)
- Shermis, M. D., "State-of-the-art automated essay scoring : Competition, results, and future directions from a United States demonstration", Assessing Writing, 20, pp.53~76 (2014)
- Shiroza, S., "Functional Variations in English: Theoretical Considerations and Practical Challenges", pp.277~293 (2020), Springer
- Snow, E. L., "Keys to detecting writing flexibility over time : entropy and natural language processing", Journal of Learning Analytics, 2(3), pp.40~54 (2015)
- Sukandi, S. S., "EFL students’ responses to learning basic reading and writing skills", Studies in English Language and Education, 5(1), pp.40~53 (2018)
- Vygotsky, L. S., "Mind in society: Development of higher psychological processes" (1978), Harvard university press
- Wang, Q., "Investigating critical thinking and knowledge construction in an interactive learning environment", Interactive Learning Environments, 17(1), pp.95~104 (2009)
- Weigle, S. C., "Teaching writing teachers about assessment", Journal of Second Language Writing, 16(3), pp.194~209 (2007)
- Winke, P., "ESL essay raters’ cognitive processes in applying the Jacobs et al. rubric: An eye-movement study", Assessing Writing, 25, pp.38~54 (2015)
- Zaytseva, V., "The effects of contexts on the acquisition of oral lexical ability in English as a foreign language", The Language Learning Journal, 49(5), pp.597~613 (2019)
