Efficacy of an Automatic Essay Scoring-based English Writing In-service Training Program
1
조회
0
다운로드
5
인용
0
참고문헌
키워드
초록
With an ever-growing demand for writing instruction in the English Language Teaching (ELT), English teachers are expected to develop teaching skills in English writing by improving their own English writing skills. This need for teaching and learning English writing skills for teachers, however, has been impeded by the logistic problem posed by the inherently time-consuming nature of the evaluation process. Thus, a wide variety of automatic essay scoring (AES) schemes have been developed to empower teachers in coping with this subjective scoring problem. The present study is intended to probe the robustness of AES-based ratings, the efficacy of an in-service training program designed to enable English teachers to develop their English writing skills with the help of an AES-based program, and the overall quality of the teaching program as perceived by participants. A comparative analysis of AES-based ratings and human ratings substantiates the robustness of the AES scheme. Overall exploration of the program’s efficacy reveals that the educational program exerted a positive impact on the subjects’ attitudes toward English writing and their acquisition of mechanical grammar. The survey demonstrates that the majority of respondents were satisfied with the overall quality of the program and the reliability of AES ratings.
Abstract
With an ever-growing demand for writing instruction in the English Language Teaching (ELT), English teachers are expected to develop teaching skills in English writing by improving their own English writing skills. This need for teaching and learning English writing skills for teachers, however, has been impeded by the logistic problem posed by the inherently time-consuming nature of the evaluation process. Thus, a wide variety of automatic essay scoring (AES) schemes have been developed to empower teachers in coping with this subjective scoring problem. The present study is intended to probe the robustness of AES-based ratings, the efficacy of an in-service training program designed to enable English teachers to develop their English writing skills with the help of an AES-based program, and the overall quality of the teaching program as perceived by participants. A comparative analysis of AES-based ratings and human ratings substantiates the robustness of the AES scheme. Overall exploration of the program’s efficacy reveals that the educational program exerted a positive impact on the subjects’ attitudes toward English writing and their acquisition of mechanical grammar. The survey demonstrates that the majority of respondents were satisfied with the overall quality of the program and the reliability of AES ratings.
참고문헌 (44)
- Attali, Y., "Exploring the feedback and revision features of Criterion", Paper presented at the National Council on Measurement in Education (NCME) (2004)
- Attali, Y., "Performance of a generic approach in automated essay scoring", The Journal of Technology, Learning, and Assessment, 10(3) (2010)
- Attali, Y., "Automated essay scoring with e-rater V.2", The Journal of Technology, Learning, and Assessment, 4(3) (2006)
- Bachman, L. F., "Fundamental considerations in language testing" (1990), Oxford University Press
- Bachman, L. F, "Language testing in practice" (1996), Oxford University Press
- Ben-Simon, A., "Toward more substantively meaningful automated essay scoring", The Journal of Technology, Learning, and Assessment, 6(1) (2007)
- Burstein, J., "Criterion online essay evaluation: an application for automated evaluation of student essays", Proceedings of the fifteenth annual conference on innovative applications of artificial intelligence (2003), The Association for the Advancement of Artificial Intelligence
- Burstein, J., "Automated essay evaluation: The Criterion online writing service", AI Magazine, 25(3) (2004), The Association for the Advancement of Artificial Intelligence
- Computer analysis of essay content for automated score prediction, "Burnstein, J.", Educational Testing Service, ~, pp.Research Report 98-15 (1998)
- Carl, J., "Errors in language learning and use: Exploring error analysis" (1998), Longman
- Cha, K. W., "An error analysis of English writing through Korean college students’ English compositions", English Teaching, 40, pp.139~157 (1990)
- ASR-based simulated oral proficiency interview model for performance testing in secondary education, "Choi, I-C", ~ (2005)
- Corder, S. P., "Idiosyncratic dialects and error analysis", International Review of Applied Linguistics, 5, pp.161~170 (1971)
- Edelblut, P, "The effects of inclusion of native speakers’ writing samples on the domain scoring accuracy of automated essay scoring of writing submitted to Taiwanese English language learners", Paper presented at the 32nd Annual Conference of the International Association for Educational Assessment (2006)
- Edelblut, P., "An analysis of IntelliMetric scoring of essays written in Bahasa Malay, Hebrew, and English and a discussion of issues related to providing multilingual scoring to responses for a single prompt", Paper presented at the 1st EAAA Conference (2005)
- Elliot, S. M., "Construct validity of Intellimetric with international assessment" (1999), Vantage Technologies
- Applying Intellimetric technology to the scoring of K-12 persuasive writing: A subsample cross validation study, "Elliot, S. M", Vantage Technologies, ~, pp.RB-424 (2000)
- Elliot, S. M., "Intellimetric: From here to validity, In Automated essay scoring: A cross-disciplinary perspective", pp.67~81 (2003), Lawrence Erlbaum
- "ETS Criterion"[URL]
- Foltz, P., "Supporting content-based feedback in on-line writing evaluation with LSA", Interactive Learning Environments, 8, pp.111~127 (2000)
- Grabe, W, "Theory and practice of writing" (1996), Addison Wesley Longman
- Kim, M. S., "An error analysis of a learner corpus of written and spoken English", 박사, pp.2010
- 김신혜, "비원어민 교사들의 영어교수학습에 대한 인식", 영어학연구, 20(20), pp.21~47 (2005), 한국영어학학회
- Kim, S. H., "Woneomin gyosa programi godeunghakgyo yeongeo gyoyuke michineun yeonghyang", English Teaching, 9(3), pp.211~237 (2002)
- Landauer, T. K., "Applications of Latent Semantic Analysis", 24th Annual Meeting of the Cognitive Science Society (2002)
- Landauer, T. K., "An introduction to Latent Semantic Analysis", Discourse Processes, 25, pp.259~284 (1998)
- Landauer, T. K, "Automatic essay assessment", Assessment in Education, 10, pp.3~ (2003)
- Lemaire, B., "Cognitive models based on Latent Semantic Analysis", Tutorial given at the 5th International Conference on Cognitive Modeling (ICCM’2003) (2003)
- Lenon, P., "Error: Some problems of definition, identification and distinction", Applied Linguistics, 12, pp.180~196 (1991)
- McNamara, T. F., "Measuring second language performance" (1996), Longman
- Micro Power, "Readability calculations (Version 7.5)" (2008), Micro Power & Light Co
- 문영인, "영어로 진행하는 영어수업(TETE)에 관한 중학교 영어 교사들의 인식 조사", 영어교육연구, 14(2), pp.299~324 (2002), 팬코리아영어교육학회
- Page, E. B., "The imminence of grading essays by computer", Phi Delta Kappan, 48, pp.238~243 (1966)
- Page, E. B., "Computer grading of student prose, using modern concepts and software", Journal of Experimental Education, 62(2), pp.27~142 (1994)
- Page, E. B., "Project essay grade: PEG. Automated essay scoring: A cross-disciplinary perspective" (2003), Lawrence Erlbaum
- Park, Chongwon, "Revisiting error analysis in English composition.", 영어교육연구, 19(1), pp.57~3 (2007), 팬코리아영어교육학회
- Richards, L., "NVivo 8" (2008), Sage Publications Software
- Rudner, L. M., "An evaluation of the IntelliMetric essay scoring system", The Journal of Technology, Learning, and Assessment, 4(4) (2006)
- Scott, M., "WordSmith 5" (2010), Lexical Analysis Software
- Sukkarieh, J. Z., "-rater: Automatic content scoring for short constructed responses", Proceedings of the Twenty-Second International FLAIRS Conference (2009)
- Sukkarieh, J. Z., "Towards agile and test-driven development in NLP applications", Proceedings of the NAACL HLT Workshop on Software Engineering, Testing, and Quality Assurance for Natural Language Processing (2009), The Association for Computational Linguistics
- Sukkarieh, J. Z., "Automating model building in C-rater", Proceedings of the 2009 Workshop on Applied Textual Inference (2009), ACL and AFNLP
- Shohamy, E., "Test impact revisited: washback effect over time", Language Testing, 13(3), pp.298~317 (1996)
- Weigle, S. C., "Assessing writing" (2002), Cambridge University Press