Engagement With Automated Writing Feedback in Mandated vs. Voluntary Conditions
2
조회
0
다운로드
2
인용
0
참고문헌
초록
This study investigates student engagement with Écree, a Natural Language Processing (NLP) based automated writing evaluation (AWE) that gives feedback on content. In particular, this study examines how mandating AWE use or not, affects students’ behavioral and emotional engagement with AWE feedback. Two classes—one in which Écree use was mandatory (EM) and another in which it was optional (EO)—were surveyed and four volunteers from each class were tracked. Overall, the two groups differed in their evaluations of how much they incorporated AWE feedback into revisions as well as in the number of uploads to Écree. Tracking the focal students’ written drafts, Écree feedback, and interview data revealed that students from EM tended to incorporate Écree’s feedback into their revisions more than EO students. Unexpectedly, EM students developed quite negative views toward Écree, whereas EO students often referred to its usefulness: its systematicity, speed, and role as a reader. This study implies that AWE can be utilized as a supplemental tool in higher education and that students may well need training so as to maximize its positive effects.
Abstract
This study investigates student engagement with Écree, a Natural Language Processing (NLP) based automated writing evaluation (AWE) that gives feedback on content. In particular, this study examines how mandating AWE use or not, affects students’ behavioral and emotional engagement with AWE feedback. Two classes—one in which Écree use was mandatory (EM) and another in which it was optional (EO)—were surveyed and four volunteers from each class were tracked. Overall, the two groups differed in their evaluations of how much they incorporated AWE feedback into revisions as well as in the number of uploads to Écree. Tracking the focal students’ written drafts, Écree feedback, and interview data revealed that students from EM tended to incorporate Écree’s feedback into their revisions more than EO students. Unexpectedly, EM students developed quite negative views toward Écree, whereas EO students often referred to its usefulness: its systematicity, speed, and role as a reader. This study implies that AWE can be utilized as a supplemental tool in higher education and that students may well need training so as to maximize its positive effects.
참고문헌 (33)
- Armel, D., "The effects of required and optional computer-based note taking on achievement and instruction completing time", Journal of Educational Computing Research, 14(4), pp.329~344 (1996)
- Attali, Y., "Exploring the feedback and revision features of Criterion", Paper presented at the National Council on Measurement in Education (2004)
- Attali, Y., "Automated essay scoring with e-rater v.2", Journal of Technology, Learning, and Assessment, 4(3), pp.1~30 (2006)
- Baron, D., "When professors get A’s and the machines get F’s", Chronicle of Higher Education, A56~ (1998)
- Brandl, K., "Effects of required and optional exchange tasks in online language learning environments", ReCALL, 24(1), pp.85~107 (2012)
- Burstein, J., "CriterionSM:Online essay evaluation: An application for automated evaluation of student essays", Proceedings of the Fifteenth Annual Conference on Innovative Applications of Artificial Intelligence, pp.3~10 (2004), AAAI Press
- "CCCC position statement on teaching, learning, and assessing writing in digital environments" (2004)[URL]
- "Writing assessment:A position statement" (2006)[URL]
- Chen, C. -F. E., "Beyond the design of automated writing evaluation : Pedagogical practices and perceived learning effectiveness in EFL writing classes", Language Learning & Technology, 12(2), pp.94~112 (2008)
- Cheville, J., "Automated scoring technologies and the rising influence of error", The English Journal, 93(4), pp.47~52 (2004)
- Elliot, S. M., "The impact of MY Access!™ use on student writing performance:A technology overview and four studies", Paper presented at the annual meeting of the American Educational Research Association (2004)
- Ericsson, P. F., "Machine scoring of human essays: Truth and consequences" (2006), Utah State University Press
- Farrell, T. S. C., "Reflective practice framework for TESOL teachers : One teacher’s reflective journey", Reflective Practice, 20(1), pp.1~12 (2019)
- "Interactive Multimedia Electronic Journal of Computer-Enhanced Learning", 1999, ~
- Garland, K., "The effects of mandatory and optional use on students’ ratings of a computer-based learning package", British Journal of Educational Technology, 35(3), pp.263~273 (2004)
- Grimes, D., "Utility in a fallible tool: A multi-site case study of automated writing evaluation", Journal of Technology, Learning, and Assessment, 8(6), pp.4~44 (2010)
- "Écree" (2018)[URL]
- Stuart Jones, "Digital Technology in the 4IR and the Future of English Learning From the Perspective of Korean EFL University Students", 현대영어교육, 20(1), pp.53~70 (2019), 현대영어교육학회[DOI]
- Lai, Y. H., "Which do students prefer to evaluate their essays: Peers or computer program?", British Journal of Educational Technology, 41(3), pp.432~454 (2010)
- Leki, I., "Feedback in second language writing: Contexts and issues", pp.266~286 (2006), Cambridge University Press
- Li, J., "Rethinking the role of automated writing evaluation (AWE) feedback in ESL writing instruction", Journal of Second Language Writing, 27(1), pp.1~18 (2015)
- Myers, M., "Automated essay scoring: A cross-disciplinary approach", pp.3~20 (2003), Lawrence Erlbaum Associates
- 박준희, "An AI-based English Grammar Checker vs. Human Raters in Evaluating EFL Learners’ Writing", 멀티미디어 언어교육, 22(1), pp.112~131 (2019), 한국멀티미디어언어교육학회[DOI]
- The association between TOEFL iBT® test scores and the Common European Framework of Reference (CEFR) levels, "Papageorgiou, S.", Educational Testing Service, ~, pp.Research Memorandum No. RM-15-06 (2015)
- Shermis, M. D., "Automated essay scoring: A cross-disciplinary perspective" (2003), Lawrence Erlbaum Associates
- Shermis, M. D., "The impact of automated essay scoring on high stakes writing assessments", Paper presented at the annual meeting of the National Council on Measurement in Education (2004)
- Shermis, M. D., "The impact of automated essay scoring on writing outcomes", Paper presented at the Annual Meeting of the National Council on Measurement in Education (2008)
- "MY Access!® efficacy report" (2007)[URL]
- Wang, Y. -J., "Exploring the impact of using automated writing evaluation in English as a foreign language university students’writing", Computer Assisted Language Learning, 26(3), pp.234~257 (2013)
- Warschauer, M., "Automated writing assessment in the classroom", Pedagogies : An International Journal, 3(1), pp.22~36 (2008)
- Warschauer, M., "Automated writing evaluation : Defining the classroom research agenda", Language Teaching Research, 10(2), pp.1~24 (2006)
- Williamson, M. M., "Validity of automated scoring : Prologue for a continuing discussion of machine scoring student writing", Journal of Writing Assessment, 1(2), pp.85~104 (2004)
- Zhang, Z., "Student engagement with computer-generated feedback : A case study", ELT Journal, 71(3), pp.317~328 (2017)