A Validation of EFL Essay Assessment Based on Corpus Indices and Error Analysis
0
조회
0
다운로드
2
인용
0
참고문헌
키워드
초록
An ever-growing demand for teaching English writing skills has been impeded by the logistic problem posed by the inherently time-consuming nature of the sophisticated process of writing assessment, whose validity and inter-rater reliability can be deteriorated by idiosyncratic human ratings. Thus, a wide variety of computer-based automatic essay scoring (AES) schemes have been developed and employed to enable language educators to cope with the issue of practicality and subjectivity in the scoring of performance tests. The first part of the present research (pilot study) is intended to probe the validity of AES-based ratings in comparison with human ratings. A comparative analysis of AES-based ratings and human ratings appears to substantiate the robustness of the AES scheme. Based on the positive results of AES, the second part of the research (main study) attempts to capitalize on quantitative analysis of corpus linguistic indices and a systematic qualitative error analysis to explore the validity of human assessment of essays collected from a writing test. The results of data analyses are provided to discuss the possibility of utilizing a corpus-based analysis in order to enhance the validity of the essay assessment process, which will lead to more effective and plausible EFL writing teaching and testing
Abstract
An ever-growing demand for teaching English writing skills has been impeded by the logistic problem posed by the inherently time-consuming nature of the sophisticated process of writing assessment, whose validity and inter-rater reliability can be deteriorated by idiosyncratic human ratings. Thus, a wide variety of computer-based automatic essay scoring (AES) schemes have been developed and employed to enable language educators to cope with the issue of practicality and subjectivity in the scoring of performance tests. The first part of the present research (pilot study) is intended to probe the validity of AES-based ratings in comparison with human ratings. A comparative analysis of AES-based ratings and human ratings appears to substantiate the robustness of the AES scheme. Based on the positive results of AES, the second part of the research (main study) attempts to capitalize on quantitative analysis of corpus linguistic indices and a systematic qualitative error analysis to explore the validity of human assessment of essays collected from a writing test. The results of data analyses are provided to discuss the possibility of utilizing a corpus-based analysis in order to enhance the validity of the essay assessment process, which will lead to more effective and plausible EFL writing teaching and testing
참고문헌 (48)
- Attali, Y., "Exploring the feedback and revision features of Criterion", Paper presented at the National Council on Measurement in Education (2004)
- Attali, Y., "Performance of a generic approach in automated essay scoring", The Journal of Technology, Learning, and Assessment, 10(3) (2010)
- Attali, Y., "Automated essay scoring with c-rater V.2", The Journal of Technology, Learning, and Assessment, 4(3) (2006)
- Bachman, L. F., "Fundamental considerations in language testing" (1990), Oxford University Press
- Bachman, L. F., "Language testing in practice" (1996), Oxford University Press
- Bachman, L. F., "Language assessment in practice" (2010), Oxford University Press
- Ben-Simon, A., "Toward more substantively meaningful automated essay scoring", The Journal of Technology, Learning, and Assessment, 6(1) (2007)
- Computer analysis of essay content for automated score prediction, "Burnstein, J.", ~, pp.Research Report 15 (1998)
- Burstein, J., "Criterion online essay evaluation: an application for automated evaluation of student essays", Proceedings of the fifteenth annual conference on innovative applications of artificial intelligence - The Association for the Advancement of Artificial Intelligence (2003)
- Burstein, J., "Automated essay evaluation: The Criterion online writing service", AI Magazine, 25(3), pp.27~35 (2004)
- Carl, J., "Errors in language learning and use : exploring error analysis" (1998), Longman
- Cha, K. W., "An error analysis of English writing through Korean college students’ English compositions", English Teaching, 40, pp.139~157 (1990)
- Chambers, F., "What do we mean by fluency?", System, 4, pp.535~544 (1997)
- 최인철, "중등영어교육 수행평가를 위한 음성인식기술기반 모의구술면접 모형", Foreign Languages Education, 12(4), pp.235~266 (2005), 한국외국어교육학회
- 최인철, "Efficacy of an Automatic Essay Scoring-based English Writing In-service Training Program", 멀티미디어 언어교육, 14(2), pp.61~89 (2011), 한국멀티미디어언어교육학회
- Corder, S. P., "Error analysis and interlanguage" (1981), Oxford Univ. Press
- "The principles of readability" (2004)[URL]
- Edelblut, P., "The effects of inclusion of native speakers’ writing samples on the domain scoring accuracy of automated essay scoring of writing submitted to Taiwanese English language learners", Paper presented at the 32nd Annual Conference of the International Association for Educational Assessment (2006)
- Edelblut, P., "An analysis of IntelliMetric scoring of essays written in Bahasa Malay, Hebrew, and English and a discussion of issues related to providing multilingual scoring to responses for a single prompt", Paper presented at the 1st EAAA Conference (2005)
- Ejzenberg, R., "The juggling act of oral fluency: A psycho-sociolinguistic metaphor, In Perspectives on fluency", pp.287~314 (2000), The University of Michigan Press
- Elliot, S. M., "Applying IntelliMetric technology to the scoring of K-12 persuasive writing: A subsample cross validation study" (2000), Vantage Technologies
- Elliot, S. M., "IntelliMetric: From here to validity, In Automated essay scoring: A cross-disciplinary perspective", pp.67~81 (2003), Lawrence Erlbaum
- Foltz, P., "Supporting content-based feedback in on-line writing evaluation with LSA", Interactive Learning Environments, 8, pp.111~127 (2000)
- Gressang, J. E., "A frequency and error analysis of the use of determiners, the relationships between noun phrases, and the structure of discourse in English essays by native English writers and native Chinese,Taiwanese, and Korean learners of English as a Second language", 박사, pp.2010
- Hieke, A. E., "A componential approach to oral fluency evaluation", Modern Language Journal, 69, pp.135~142 (1985)
- 김미숙, "(An)Error analysis of a learner corpus of written and spoken English produced by Korean University students", 박사, pp.2010
- Kim, Y. J., "Writing a coherent text in a foreign language: Analysis of Korean EFL students’ argumentative essays", 박사, pp.2011
- Landauer, T. K., "Applications of latent semantic analysis", Proceedings of the 24th annual meeting of the Cognitive Science Society (2002)
- Landauer, T. K., "An introduction to latent semantic analysis", Discourse Processes, 25, pp.259~284 (1998)
- Landauer, T. K, Laham, D., "Automatic essay assessment", Assessment in Education, 10, pp.3~ (2003)
- Lemaire, B. & Denhire, G., "Cognitive models based on Latent Semantic Analysis", Tutorial given at the 5th International Conference on Cognitive Modeling (ICCM’2003) (2003)
- Lennon, P., "Investigating fluency in EFL: A quantitative approach", Language Learning, 40(3), pp.387~417 (1990)
- Leograndis, D., "Fluent writing" (2006), NH, Heinemann
- McNamara, T. F., "Measuring second language performance" (1996), Longman
- Micro Power, "Readability calculations" (2008), Micro Power & Light Co
- Murayama, T., "Native speakers’ responses to non-native speakers’ sociolinguistic errors in Japanese", 석사, pp.1996
- Oller, J. W. Jr, "Review of content and construct validation of a criterion-referenced English proficiency test by Choi(1994)", English Teaching, 50(3), pp.161~68 (1995)
- Page, E. B., "The imminence of grading essays by computer", Phi Delta Kappan, 48, pp.238~243 (1966)
- Page, E. B., "Computer grading of student prose, using modern concepts and software", Journal of Experimental Education, 62(2), pp.27~142 (1994)
- Page, E. B., "Project essay grade : PEG. Automated essay scoring : A cross-disciplinary perspective" (2003), Lawrence Erlbaum
- Park, Chongwon, "Revisiting error analysis in English composition.", 영어교육연구, 19(1), pp.57~3 (2007), 팬코리아영어교육학회
- Richards, L., "NVivo 8" (2009), Sage publications software
- Rudner, L. M., "An evaluation of the IntelliMetric essay scoring system", The Journal of Technology, Learning, and Assessment, 4(4) (2006)
- Scott, M., "WordSmith 5" (2010), Lexical analysis software
- Sukkarieh, J. Z., "c-rater: Automatic content scoring for short constructed responses", Proceedings of the Twenty-Second International FLAIRS Conference (2009)
- Sukkarieh, J. Z., "Towards agile and test-driven development in NLP applications", Proceedings of the NAACL HLT Workshop on Software Engineering, Testing, and Quality Assurance for Natural Language Processing (2009)
- Sukkarieh, J. Z., "Automating model building in c-rater", Proceedings of the2009 ACL and AFNLP Workshop on Applied Textual Inference (2009)