Examining the Accuracy of Computer Scoring of Korean EFL Students' Essays
1
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
The purpose of the current study is to examine the accuracy of an automated essay scoring (AES) program, IntelliMetric, in the EFL (English as a Foreign Language) context of Korea by means of comparing the computer and human scoring of Korean EFL students' short essays in English. The data for this study are essay scores for 218 students from three high schools in Korea. The student essays are in response to two writing topics. For the purpose of this study, the means, agreement rates, and Pearson correlations between the human scores and computer scores are computed and then compared. The data analyses show that IntelliMetric, the AES program evaluated in the current study, is able to accurately score Korean EFL students' responses written in English. As a result, the comparison of means, agreement rates, and Pearson correlations between the human scores and IntelliMetric scores demonstrates that IntelliMetric is capable of scoring Korean EFL students' responses at least as accurately as trained human raters.
Abstract
The purpose of the current study is to examine the accuracy of an automated essay scoring (AES) program, IntelliMetric, in the EFL (English as a Foreign Language) context of Korea by means of comparing the computer and human scoring of Korean EFL students' short essays in English. The data for this study are essay scores for 218 students from three high schools in Korea. The student essays are in response to two writing topics. For the purpose of this study, the means, agreement rates, and Pearson correlations between the human scores and computer scores are computed and then compared. The data analyses show that IntelliMetric, the AES program evaluated in the current study, is able to accurately score Korean EFL students' responses written in English. As a result, the comparison of means, agreement rates, and Pearson correlations between the human scores and IntelliMetric scores demonstrates that IntelliMetric is capable of scoring Korean EFL students' responses at least as accurately as trained human raters.
참고문헌 (32)
- Bachman, L. F, "Language Testing in Practice" (1996), Oxford University Press
- Bachman, L. F, "Investigating Variability in Tasks and Rater Judgements in a Performance Test of Foreign Language Speaking", Language Testing, 12, pp.238~257 (1996)
- Bridgman, B, "Survey of Academic Writing Tasks", Written Communication, 1, pp.247~280 (1984)
- Burstein, J, "The E-rater Scoring Engine: Automated Essay Scoring with Natural Language Processing, In Automated Essay Scoring: A Cross Disciplinary Approach", pp.113~121 (2003), Lawrence Erlbaum Association
- Burstein, J, "Automated Essay Scoring for Nonnative English Speakers", Proceedings of the ACL '99 Workshop on Computer-Mediated Language Assessment and Evaluation of Natural Language Processing, College Park, MD (1999)
- Burstein, J, "Criterion: Online Essay Evaluation: An Application for Automated Evaluation of Student Essays", Proceedings of the 25th Annual Conference on Innovative Applications of Artificial Intelligence. Acapulco, Mexico (2003)
- Burstein, J, "Computer Analysis of Essays", Proceedings of the NCME Symposium on Automated Scoring. Montreal, Canada (1998)
- Burstein, J, "Automated Evaluation of Essays and Short Answers", Proceeding of the 5th International Computer Assisted Conference (2001), Loughborough University
- Benefits of Modularity in an Automated Essay Scoring System, "Burstein, J", ERIC, ~, pp.Reproduction Service No 032 101 (2000)
- Canesco, G, "Writing Required in Graduate Courses in Business Administration", TESOL Quarterly, 23, pp.305~316 (1989)
- Methodological Approaches to Online Scoring of Essays, "Chung, K", ERIC, ~, pp.Reproduction Service No 418 101 (1999)
- Elliot, S, "Construct Validity of IntelliMetric with International Assessment" (1999), Vantage Learning
- Elliot, S, "About IntelliMetric" (2001), Vantage Learning
- Elliot, S, "Applying IntelliMetric Technology to the Scoring of 3rd and 8th Grade Standardized Writing Assessments" (2001), Vantage Learning
- Elliot, S, "A Study of Expert Scoring, Standard Human Scoring and IntelliMetric Scoring Accuracy for Statewide Eighth Grade Writing Responses" (2002), Vantage Learning
- Elliot, S, "A True Score Study of 11th Grade Student Writing Responses Using IntelliMetric Version 9.0" (2003), Vantage Learning
- Elliot, S, "How Does IntelliMetric Score Essay Responses" (2003), Vantage Learning
- Elliot, S, "IntelliMetric: From Here to Validity, In Automated Essay Scoring: A Cross Disciplinary approach", pp.71~86 (2003), Lawrence Erlbaum Association
- Foltz, W, "Latent Sematic Analysis for Text-based Research", Behavior Research Methods, Instruments and Computers, 28(2), pp.197~202 (1996)
- "Automated Essay Scoring: Applications to Educational Technology" (1999)[URL]
- Horowitz, D, "Student Beliefs about Language Learning, In Learner Strategies in Language Learning", pp.119~129 (1987), Prentice Hall
- "Beyond Automated Essay Scoring" (2000)[URL]
- "The Intelligent Essay Grader" (2000)[URL]
- Landauer, K, "Automated Essay Scoring and Annotation of Essays with the Intelligent Essay Assessor, In Automated Essay Scoring: A Cross Disciplinary approach", pp.87~112 (2003), Lawrence Erlbaum Association
- Leki, I, "Students' Perception of EAP Writing Instruction and Writing Needs Across the Disciplines", TESOL Quarterly, 28, pp.81~101 (1994)
- Lemaire, B, "A System to Assess the Semantic Content of Student Essays", Educational Computing Research, 24(3), pp.305~326 (2001)
- Myers, M, "What Can Computers and AES Contribute to a K-12 Writing Program?, In Automated Essay Scoring: A Cross Disciplinary Approach", pp.3~20 (2003), Lawrence Erlbaum Association
- Page, E, "Project Essay Grader: PEG, In Automated Essay Scoring: A Cross Disciplinary Approach", pp.43~54 (2003), Lawrence Erlbaum Association
- An Overview of Three Approaches to Scoring Written Essays by Computer, "Runder, L", ERIC, ~, pp.Digest No ED 458 290 (2001)
- Exit Assessments: Evaluating Writing Ability Through Automated Essay Scoring, "Shermis, M", ERIC, ~, pp.Document Reproduction Service No ED 464 950 (2002)
- "The Credible Grading Machine: Essay Scoring in the DOD" (2004)[URL]
- Suen, H. K, "Principles of Test Theory" (1990), Lawrence Erlbaum Associates