Investigating the feasibility of generic scoring models of e-rater for TOEFL iBT independent writing tasks
0
조회
0
다운로드
4
인용
0
참고문헌
키워드
초록
The current study reports the findings from Phase 2 of a larger research study undertaken toinvestigate the feasibility of using generic scoring models for e-rater in the context of scoringessays for independent writing tasks for TOEFL CBT and TOEFL iBT. In Phase 1, sixdifferent variants of generic and hybrid scoring models of e-rater were created based ontransformed writing data from three different samples of TOEFL CBT prompts (n1=20, n2=20,n3= 40) with the help of ETS (Educational Testing Service) staff and then evaluated on aseparate sample of seven TOEFL CBT prompts (Lee, 2016). In the present investigation, thesesix generic/hybrid models were used, along with prompt-specific models, to score a total of3,126 essays written for two TOEFL iBT independent writing tasks from a field study and theirperformance was evaluated. Results of the analysis showed that (a) there were relatively smallscore variations among different automated scoring models and (b) similar levels of scoreagreement were achieved between the human-human rater pair and various human-automatedrater pairs, although the prompt-specific model behaved most similarly to the human raters. Interms of criterion-related validity of scores, the human rater scores turned out to be somewhatbetter indicators of test-takers’ overall ESL (English as a Second Language) languageproficiency than the automated scores in general. Nevertheless, the comparative advantage invalidity of human rater scores (over automated scores) seemed to diminish significantly, whenmore direct writing measures, such as scores for TOEFL CBT independent writing tasks, wereused as criterion measures.
Abstract
The current study reports the findings from Phase 2 of a larger research study undertaken toinvestigate the feasibility of using generic scoring models for e-rater in the context of scoringessays for independent writing tasks for TOEFL CBT and TOEFL iBT. In Phase 1, sixdifferent variants of generic and hybrid scoring models of e-rater were created based ontransformed writing data from three different samples of TOEFL CBT prompts (n1=20, n2=20,n3= 40) with the help of ETS (Educational Testing Service) staff and then evaluated on aseparate sample of seven TOEFL CBT prompts (Lee, 2016). In the present investigation, thesesix generic/hybrid models were used, along with prompt-specific models, to score a total of3,126 essays written for two TOEFL iBT independent writing tasks from a field study and theirperformance was evaluated. Results of the analysis showed that (a) there were relatively smallscore variations among different automated scoring models and (b) similar levels of scoreagreement were achieved between the human-human rater pair and various human-automatedrater pairs, although the prompt-specific model behaved most similarly to the human raters. Interms of criterion-related validity of scores, the human rater scores turned out to be somewhatbetter indicators of test-takers’ overall ESL (English as a Second Language) languageproficiency than the automated scores in general. Nevertheless, the comparative advantage invalidity of human rater scores (over automated scores) seemed to diminish significantly, whenmore direct writing measures, such as scores for TOEFL CBT independent writing tasks, wereused as criterion measures.
참고문헌 (26)
- Attali, Y., "Performance of a generic approach in automated essay scoring", Journal of Technology, Learning, and Assessment, 10(3), pp.1~6 (2010)
- Attali, Y., "Automated essay scoring with e-rater V.2", Journal of Technology, Learning and Assessment, 4(3), pp.3~30 (2006)
- Attali, Y., "Validity of scores for a developmental writing scale based on automated scoring", Educational and Psychological Measurement, 69, pp.978~993 (2009)
- Moving the field forward: Some thoughts on validity and automated scoring, "Bennett, R. E.", ETS, ~, pp.ETS Research Memorandum RM-04-01 (2004)
- Bennett, R. E., "Validity and automated scoring: It’s not only the scoring", Educational Measurement: Isues and Practice, 17(4), pp.9~17 (1998)
- Burstein, J., "Automated essay scoring: A cross-disciplinary perspective", pp.113~121 (2003), Lawrence Erlbaum
- Burstein, J., "Automated Essay scoring for Nonnative English Speakers", Paper presented at the Joint Symposium of the Association of Computational Linguistics and the International Association of Language Learning Technologies, Workshop on Computer-Mediated Language Assessment and Evaluation of Natural Language Processing held in College Park (1999)
- Cohen, J., "A coefficient of agreement for nominal scales", Educational and Psychological Measurement, 20(1), pp.37~46 (1960)
- Beyond essay length: Evaluating e-rater’s performance on TOEFL essays, "Chodorow, M.", ETS, ~, pp.TOEFL Research Report No. 73; ETS RR-04- 04 (2004)
- Dikli, S., "An overview of automated scoring of essays", Journal of Technology, Learning, and Assessment, 5(1), pp.1~35 (2006)
- Computer-based TOEFL score user guide, "Educational Testing Service", Educational Testing Service, ~ (1998)
- TOEFL iBT Tips: How to prepare for the TOEFL iBT, "Educational Testing Service", Educational Testing Service, ~ (2007)
- Enright, M. K., "Complementing human judgment of essays written by English language learners with e-rater® scoring", Language Testing, 27(3), pp.317~334 (2010)
- Henning, G., "A guide to language testing: Development, evaluation, and research" (1987), Hienle & Hienle
- Kukich, K., "Beyond automated essay scoring", IEEE Intelligent Systems, 15(5), pp.22~27 (2000)
- 이용원, "Reliability and Validity of Automated Essay Scores for Independent Writing Tasks: Generic, Hybrid, and Prompt-Specific Models", 영미연구, 36, pp.249~277 (2016), 영미연구소
- E-rater research agenda and work plans for the next generation TOEFL, "Lee, Y. -W.", ETS, ~ (2004)
- Lee, Y. -W., "Toward automated multi-trait scoring of essays:Investigating relationships among holistic, analytic, and text feature scores", Applied Linguistics, 31(3), pp.391~417 (2010)
- Lee, Y. -W., "Evaluating prototype tasks and alternative rating schemes for a new ESL writing test through G-theory", International Journal of Testing, 7(4), pp.353~385 (2007)
- Analysis of e-rater performance on NGT writing samples, "Lee, Y. -W.", ETS, ~ (2004)
- Comparing the validity of automated and human essay scoring, "Powers, D. E.", ETS, ~, pp.GRE No. 98-08 (2000)
- Evaluation of the e-rater® scoring engine for the TOEFL® independent and integrated prompts, "Ramineni, C.", ETS, ~, pp.ETS RR-12-06 (2012)
- Rudner, L., "An overview of three approaches to scoring written essays by computer", Practical Assessment, Research & Evaluation, 7(26) (2001)
- Shermis, M. D., "State-of-the-art automated essay scoring: Competition, results, and future directions from a United States demonstration", Assessing Writing, 20, pp.53~76 (2014)
- Shermis, M. D., "Automated essay scoring: A cross-disciplinary perspective", pp.113~121 (2003), Lawrence Erlbaum
- Weigle, S., "Validation of automated scores of TOEFL iBT tasks against non-test indicators of writing ability", Language Testing, 27(3), pp.335~353 (2010)