Construct Validity in Human Scoring and Criterion: What Criterion would/would not Measure
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
The current investigation is a study which aims to examine the reliability of automated essay scoring(AES) and investigate the validity of writing (sub)constructs that Criterion would/would not measure. Criterion evaluated test-takers’ essays written for assessed iBT TOEFL independent writing tasks by comparing human raters’ evaluation. In particular, the current study explored which essay features were most closely related to each of the six different analytic dimensions of Criterion. Five types of prompts were employed to create a writing test administered to fifty college students in Seoul. The result showed that the agreement between human-rater and Criterion was moderate. In addition, three essay features(development, organization, and grammar/usage) were crucial factors to predict the holistic score in human rating while organization also was the most powerful predictor of AES overall scores. Also, mechanics and sentence variety/constructions were the second and the third strong factors to predict writing scores in AES. This factor discrepancy in measuring writing scores might reflect that a few sentence constructions were differently evaluated between Criterion and human raters. This result suggests that the feature dimensions in Criterion need refinement in its construct dimensions. The findings have some implications in teaching process writing to students and in using AES.
Abstract
The current investigation is a study which aims to examine the reliability of automated essay scoring(AES) and investigate the validity of writing (sub)constructs that Criterion would/would not measure. Criterion evaluated test-takers’ essays written for assessed iBT TOEFL independent writing tasks by comparing human raters’ evaluation. In particular, the current study explored which essay features were most closely related to each of the six different analytic dimensions of Criterion. Five types of prompts were employed to create a writing test administered to fifty college students in Seoul. The result showed that the agreement between human-rater and Criterion was moderate. In addition, three essay features(development, organization, and grammar/usage) were crucial factors to predict the holistic score in human rating while organization also was the most powerful predictor of AES overall scores. Also, mechanics and sentence variety/constructions were the second and the third strong factors to predict writing scores in AES. This factor discrepancy in measuring writing scores might reflect that a few sentence constructions were differently evaluated between Criterion and human raters. This result suggests that the feature dimensions in Criterion need refinement in its construct dimensions. The findings have some implications in teaching process writing to students and in using AES.
참고문헌 (28)
- Attali, Y., "Exploring the feedback and revision features of Criterion", Paper presented at the National Council on the measurement in education (2004)
- Attali, Y., "Handbook of automated essay evaluation: Current applications and new directions", pp.181~198 (2013), Routledge
- Burstein, J., "Automated Essay Scoring for Nonnative English Speakers", Proceeding ASSESSEVALNLP '99Proceedings of a Symposium on Computer Mediated Language Assessment and Evaluation in Natural Language Processing, pp.68~75 (1999)
- Burstein, J., "Automated Essay Evaluation: The Criterion Online Writing Service", AI Magazine, 25(3), pp.27~36 (2004)
- Beyond Essay Length: Evaluating e-rater®’s Performance on TOEFL Essays, "Chodorow, M.", ETS, ~, pp.ETS report, RR-04-04 (2004)
- 최재호, "영어작문의 과정적 글쓰기에서 자동작문평가(Automated Essay Scoring)의 효과", 교육정보미디어연구, 17(2), pp.177~196 (2011), 한국교육정보미디어학회
- Cohen, Y., "Validating human and automated scoring of essays against"True"scores", Applied measurement in education, 31(3), pp.241~250 (2018)
- Ebyary, K., "The Impact of Computer-based Feedback on Students’ written work", International Journal of English Studies, 10(2), pp.121~142 (2010)
- Enright, M. K., "Complementing human judgment of essays written by English language learners with e-rater® scoring", Language Testing, 27(3), pp.317~334 (2010)
- James, C., "Validating a Computerized Scoring system of Assessing Writing and Placing students in composition courses", Assessing Writing, 11(3), pp.167~178 (2006)
- 김나영, "Different Chat Modes of a Chatbot and EFL Students’ Writing Skills Development", 외국어교육연구, 32(1), pp.263~290 (2018), 외국어교육연구소[DOI]
- 김태은, "Reflection on Using the Criterion Online Writing Evaluation Service", 멀티미디어 언어교육, 13(3), pp.59~83 (2010), 한국멀티미디어언어교육학회[DOI]
- 고원영, "Effective Applications of Automated Writing Feedback in Process-based Writing Instruction", 영어교육, 72(3), pp.91~118 (2017), 한국영어교육학회[DOI]
- 이상민, "Exploring the Potential of a Web-Based Writing Instruction Program and AES: An Empirical Study Using My Access", 멀티미디어 언어교육, 11(2), pp.103~125 (2008), 한국멀티미디어언어교육학회[DOI]
- 이영주, "영작문 시간에 활용된 자동 쓰기 평가 피드백에 대한 학생들의 인식 탐색", 중등영어교육, 10(4), pp.143~164 (2017), 한국중등영어교육학회
- 이용원, "Investigating the feasibility of generic scoring models of e-rater for TOEFL iBT independent writing tasks", 영어교육연구, 28(1), pp.101~122 (2016), 팬코리아영어교육학회[DOI]
- Analytic Scoring of TOEFL®CBT Essays: Scores from Humans and E-rater®, "Lee, Y. W.", ETS, ~, pp.ETS Report, RR-08-01 (2008)
- Lee, Y. W., "Toward Automated Multi-trait Scoring of Essays : Investigating Relationships among Holistic, Analytic, and Text Feature Scores", Applied Linguistics, 31, pp.391~417 (2010)
- Li, Z., "The Role of Automated Writing Evaluation Holistic Scores in the ESL classroom", System, 44, pp.66~78 (2014)
- Mohsen, M. A., "The Effectiveness of Using a Hybrid Mode of Automated Writing Evaluation System on EFL Students’ Writing", Teaching English with Teaching, 19(1), pp.118~131 (2019)
- 문영인, "Short-term Effects of Automated Writing Feedback and Users’ Evaluation of Criterion", 응용언어학, 27(4), pp.125~150 (2011), 한국응용언어학회
- 박태준, "Examining the Accuracy of Computer Scoring of Korean EFL Students' Essays", 언어과학연구, (56), pp.53~74 (2011), 언어과학회
- Powers, D., "Comparing the validity of automated and human essay scores", Journal of Educational Computing Research, 26(4), pp.407~425 (2002)
- Shi, B., "Effect of prompt type on test-takers’writing performance and writing strategy use in the continuation task", Language Testing, 37(3), pp.361~388 (2020)
- Stevenson, M., "The effects of computer-generated feedback on the quality of writing", Assessing Writing, 19, pp.91~65 (2014)
- Wang, J., "Automated Essay Scoring versus Human Scoring: a comparative study", Journal of Technology, Learning and Assessment, 6(2), pp.719~725 (2007)
- Weigle, S., "Validation of Automated Scores of TOEFL iBT tasks against non-test indicators of writing ability", Language Testing, 27(3), pp.335~353 (2010)
- 윤도현, "How Do On-Line Students Use Technology to Enhance Their Learning?", 외국어교육연구, 28(1), pp.51~67 (2014), 외국어교육연구소[DOI]