Exploring Automated Scoring of Korean High School Students’ English Composition
0
조회
0
다운로드
2
인용
0
참고문헌
키워드
초록
The primary purpose of this study was to explore the efficacy of an automated essay scoring (AES) program, namely IntelliMetric, in scoring Korean high school students’ English compositions in a large-scale standardized writing assessment setting. In addition, this study also examined the relative utility of 5-point analytic rating scales and 4-point analytic rating scales for use in scoring essays from this particular group of secondary EFL students in Korea. The results of this study showed that the computer scoring models built based on a large number of student essays with known scores (i.e., training set) could successfully predict the scores of essays set aside for the purpose of internal validation (i.e., validation set), providing evidence for the efficacy of the computer scoring models developed. This was backed up by three different measures comparing the scores derived from the automated essay scoring program and trained human raters: mean scores, agreement rates, and pearson correlation. Considering the computer program’s ability to apply scoring criteria or standards consistently over time, although far from being perfect, it seems to show the potential to score essays reliably large scale assessment contexts such as a nation-wide writing test.
Abstract
The primary purpose of this study was to explore the efficacy of an automated essay scoring (AES) program, namely IntelliMetric, in scoring Korean high school students’ English compositions in a large-scale standardized writing assessment setting. In addition, this study also examined the relative utility of 5-point analytic rating scales and 4-point analytic rating scales for use in scoring essays from this particular group of secondary EFL students in Korea. The results of this study showed that the computer scoring models built based on a large number of student essays with known scores (i.e., training set) could successfully predict the scores of essays set aside for the purpose of internal validation (i.e., validation set), providing evidence for the efficacy of the computer scoring models developed. This was backed up by three different measures comparing the scores derived from the automated essay scoring program and trained human raters: mean scores, agreement rates, and pearson correlation. Considering the computer program’s ability to apply scoring criteria or standards consistently over time, although far from being perfect, it seems to show the potential to score essays reliably large scale assessment contexts such as a nation-wide writing test.
참고문헌 (31)
- Bachman, L. F., "Some reflections on task-based language performance assessment", Language Testing, 19, pp.453~476 (2002)
- Bachman, L. F., "Investigating variability in tasks and rater judgements in a performance test of foreign language speaking", Language Testing, 12, pp.238~257 (1995)
- Bachman, L. F., "Language Testing in Practice" (1996), Oxford University Press
- Bonk, W. J., "A many-facet Rasch analysis of the second language group oral discussion task", Language Testing, 20, pp.89~110 (2003)
- Burstein, J., "The e-rater scoring engine: Automated essay scoring with natural language processing, In Automated essay scoring: A cross disciplinary approach", pp.113~121 (2003), Lawrence Erlbaum Association
- Burstein, J., "Automated essay scoring for nonnative English speakers", Proceedings of the ACL ‘99 Workshop on Computer-Mediated Language Assessment and Evaluation of Natural Language Processing (1999), College Park
- Burstein, J., "Criterion: Online essay evaluation: An application for automated evaluation of student essays", Proceedings of the 25th Annual Conference on Innovative Applications of Artificial Intelligence (2003), Acapulco
- Burstein, J., "Computer analysis of essays", Proceedings of the NCME Symposium on Automated Scoring (1998), Montreal
- 최인철, "Efficacy of an Automatic Essay Scoring-based English Writing In-service Training Program", 멀티미디어 언어교육, 14(2), pp.61~89 (2011), 한국멀티미디어언어교육학회
- Chung, K., "Methodological approaches to online scoring of essays", ERIC Reproduction Service No. ED 418 101, ~ (1997)
- Elliot, S., "Construct validity of IntelliMetric with international assessment" (1999), Vantage Learning
- A study of expert scoring and IntelliMetric scoring accuracy for dimensional scoring of grade 11 student writing responses, "Elliot, S.", Vantage Learning., ~, pp.RB-397 (2000)
- A true score study IntelliMetric accuracy for holistic and dimensional scoring of college entry-level writing program, "Elliot, S.", Vantage Learning., ~, pp.RB-407 (2000)
- Elliot, S., "About IntelliMetric" (2001), Vantage Learning
- Elliot, S., "Applying IntelliMetric technology to the scoring of 3rd and 8th grade standardized writing assessments" (2001), Vantage Learning
- Elliot, S., "A study of expert scoring, standard human scoring and IntelliMetric scoring accuracy for statewide eighth grade writing responses" (2002), Vantage Learning
- Elliot, S., "A true score study of 11th grade student writing responses using IntelliMetric version 9. 0." (2003), Vantage Learning
- Elliot, S., "How does IntelliMetric score essay responses?" (2003), Vantage Learning
- Elliot, S., "IntelliMetric: From here to validity, In Automated essay scoring: A cross disciplinary approach", pp.71~86 (2003), Lawrence Erlbaum Association
- Foltz, W., "Latent sematic analysis for text-based research. Behavior Research Methods", Instruments and Computers, 28(2), pp.197~202 (1996)
- "Automated essay scoring: Applications to educational technology" (1999)[URL]
- Kondo-Brown, K., "A FACETS analysis of rater bias in measuring Japanese second language writing performance", Language Testing, 19, pp.3~31 (2002)
- Landauer, K., "Automated essay scoring and annotation of essays with the Intelligent Essay Assessor, In Automated essay scoring: A cross disciplinary approach", pp.87~112 (2003), Lawrence Erlbaum Association
- Lemaire, B., "A system to assess the semantic content of student essays", Educational Computing Research, 24(3), pp.305~326 (2001)
- Lumley, T., "Assessment criteria in a large-scale writing test: What do they really mean to the raters?", Language Testing, 19, pp.246~276 (2002)
- Lynch, B. K., "Using G-theory and many-facet Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants", Language Testing, 15, pp.158~180 (1998)
- McNamara, T. F., "Measuring second language performance" (1996), Longman
- Myers, M., "What can computers and AES contribute to a K-12 writing program?, In Automated essay scoring: A cross disciplinary approach", pp.3~20 (2003), Lawrence Erlbaum Association
- Page, E., "Project essay grader: PEG, In Automated essay scoring: A cross disciplinary approach", pp.43~54 (2003), Lawrence Erlbaum Association
- Suen, H. K., "Principles of test theory" (1990), Lawrence Erlbaum Associates
- Weigle, S. C., "Integrating reading and writing in a competency test for non-native speakers of English", Assessing Writing, 9, pp.27~55 (2004)