본문 바로가기
KERD
논문KCI 등재

영어작문수업에서의 자동작문채점기 FES 활용에 대한타당성 연구

Validation study of automated essay scoring system applicable to English writing class

이희경|황명환
팬코리아영어교육학회 (Pan-Korea English Teachers Association)|2015.09|영어교육연구 (English Language Teaching)|제27권 제3호|pp. 261-283|DOI 10.17936/pkelt.2015.27.3.013

1

조회

0

다운로드

5

인용

0

참고문헌

키워드

validation타당도 검증과제 난이도task difficultygeneralizability theory일반화가능도이론자동작문채점기FES
주제분류영어와문학

초록

This study aims to validate automated essay scoring(AES) named free essay scoring(FES), inorder to explore its applicability in English writing classes in Korea. To achieve this aim, a totalof 57 essays from 27 high school students were selected and upon the completion of essaywriting, a retrospective interview was conducted. Further the grading results of AES werecompared with those of human raters. The analysis using Generalizability theory revealed thatthe grading results through AES were stable by presenting generalizability coefficient relativelyrobust enough to apply to low stake testing situations, and the two writing tasks provided by theAES proved to be valid in terms of the task difficulty as intended by the system. That is, avariety of evidences investigated in this research showed that the writing tasks the AESprovided were classified according to the task difficulty. Lastly, students showed somewhatmixed perceptions on the use of AES and its feedback. Further implications were suggested.

Abstract

This study aims to validate automated essay scoring(AES) named free essay scoring(FES), inorder to explore its applicability in English writing classes in Korea. To achieve this aim, a totalof 57 essays from 27 high school students were selected and upon the completion of essaywriting, a retrospective interview was conducted. Further the grading results of AES werecompared with those of human raters. The analysis using Generalizability theory revealed thatthe grading results through AES were stable by presenting generalizability coefficient relativelyrobust enough to apply to low stake testing situations, and the two writing tasks provided by theAES proved to be valid in terms of the task difficulty as intended by the system. That is, avariety of evidences investigated in this research showed that the writing tasks the AESprovided were classified according to the task difficulty. Lastly, students showed somewhatmixed perceptions on the use of AES and its feedback. Further implications were suggested.

참고문헌 (42)

  1. 강승호, "신뢰도(Reliability)" (2004), 교육과학사
  2. 김보라, "일반화가능도 이론을 적용한 초등학교 쓰기 수행평가의 총체적 채점과 분석적 채점 방식 비교", 교육학연구, 50(4), pp.49~76 (2012), 한국교육학회
  3. 김연희, "쓰기 수행 평가의 채점 기준 변이성이 영작문 평가에 미치는 영향", 영어학, 13(3), pp.465~496 (2013), 한국영어학회
  4. 신동일, "일반화가능도 이론 적용을 중심으로 한 말하기 평가도구 타당도 검증연구", 응용언어학, 17(1), pp.199~221 (2001)
  5. 양지승, "일반화가능도 이론을 적용한단위검사 구성 검사점수의 신뢰도 추정", 교육평가연구, 20(1), pp.119~139 (2007), 한국교육평가학회
  6. 임황규, "대규모 영어 쓰기 평가를 위한 에세이형 자동채점 시스템 타당화", 멀티미디어 언어교육, 17(1), pp.84~105 (2014), 한국멀티미디어언어교육학회
  7. 이종성, "일반화 가능도 이론의 연구과제", 연세교육과학, 45, pp.1~15 (1997)
  8. 최재호, "영어작문의 과정적 글쓰기에서 자동작문평가(Automated Essay Scoring)의 효과", 교육정보미디어연구, 17(2), pp.177~196 (2011), 한국교육정보미디어학회
  9. Attali, Y., "Automated essay scoring with e-rater V.2", Journal of Technology, Learning, and Assessment(JTLA), 4(3) (2006)
  10. Bachman, L. & Palmer, A., "Language assessment in practice" (2003), Oxford University Press
  11. Validity and Automated Essay Scoring. A summary paper(NI 085541), "Bennett, R. E.", Educational Testing Service, ~ (2013)
  12. Bourwer, R., "Effect of genre on the generalizability of writing scores", Language Testing, 32(1), pp.83~100 (2015)
  13. Brennan, R. L., "Generalizability theory" (2001), Springer-Verlag Berlin Heidelberg
  14. Burstein, J. C., "Automated essay scoring: A cross-disciplinary perspective", pp.113~121 (2003), Lawrence Erlbaum Associates
  15. Chen, C. F. E., "Beyond the design of automated writing evaluation : Pedagogical practices and perceived learning effectiveness in EFL writing classes", Language Learning & Technology, 12(2), pp.94~112 (2008)
  16. 최인철, "Efficacy of an Automatic Essay Scoring-based English Writing In-service Training Program", 멀티미디어 언어교육, 14(2), pp.61~89 (2011), 한국멀티미디어언어교육학회
  17. Gregory, K. W., "Automated essay scoring: A cross-disciplinary perspective", pp.23~40 (2003), Lawrence Erlbaum Associates
  18. Chung, K., "Methodological approaches to online scoring of essays", ERIC reproduction service No. ED 418 101, ~ (1997)
  19. Deremer, M. L., "Writing assessment : Raters' elaboration of the rating task", ASSESSING WRITING, 5(1), pp.7~29 (1998)
  20. Dikli, S., "An overview of automated scoring of essays", The Journal of Technology, Learning, and Assessmen, 5(1) (2006)
  21. Elliot, S., "Automated Essay Scoring: A cross-disciplinary perspective", pp.71~86 (2003), Lawrence Erlbaum Associates
  22. Ellis, R., "Task-based language learning and teaching" (2003), Oxford University
  23. Enright, M. K., "Complementing human judgment of essays written by English language learners with e-rater scoring", Language Testing, 27, pp.317~334 (2010)
  24. Sentence structure used by superior students in grade four and twelve and superior adults, "Hunt, K. W.", Cooperative Research Project. Washington D.C, U.S. Office of Education, ~ (1967)
  25. Kane, M., "Validating measures of performance", Educational Measurement : Issues and Practice, 18(2), pp.5~17 (1999)
  26. 김지은, "The Effectiveness of Automated Essay Scoring in an EFL College Classroom", 멀티미디어 언어교육, 17(3), pp.11~36 (2014), 한국멀티미디어언어교육학회
  27. Kondo-Brown, K., "A facets analysis of rater bias in measuring Japanese second language writing performance", Languages Testing, 19, pp.3~31 (2002)
  28. Landauer, T. K., "Automated essay scoring: A cross-disciplinary perspective", pp.87~112 (2003), Lawrence Erlbaum Associates
  29. 이상민, "Exploring the Potential of a Web-Based Writing Instruction Program and AES: An Empirical Study Using My Access", 멀티미디어 언어교육, 11(2), pp.103~125 (2008), 한국멀티미디어언어교육학회
  30. 맹은경, "The Effect and Teachers' Perception of Using an Automated Essay Scoring System in L2 Writing", 영어학연구, 16(1), pp.247~275 (2010), 한국영어학학회
  31. McNamara, T. F., "Measuring second language performance" (1996), Longman
  32. Myers, M., "Automated essay scoring: A cross-disciplinary perspective", pp.3~20 (2003), Lawrence Erlbaum Associates
  33. Page, E. B., "Computer grading of student prose, using modern concepts and software", Journal of Experimental Education, 62, pp.127~142 (1994)
  34. 박묘영, "Exploring the Raters’ Bias on an EFL Writing Assessment Using Multi-faceted Rasch Measurement", Studies in English Education, 17(2), pp.175~202 (2012), 글로벌영어교육학회
  35. Robinson, P., "The Cognition Hypothesis of adult, task-based language learning", Second Language Studies, 21, pp.45~107 (2003)
  36. Schaefer, E., "Rater bias patterns in an EFL writing assessment", Language Testing, 25(4), pp.465~493 (2008)
  37. Shaw, S. D., "Examining writing" (2007), Cambridge University Press
  38. Shermis, M., "Exit assessments: Evaluating writing ability through automated essay scoring", (ERIC document reproduction service no ED 464 950, ~ (2002)
  39. Skehan, P., "A cognitive approach to language learning" (2008), Oxford University Press
  40. Wang, Y. J., "Exploring the impact of using automated writing evaluation in English as a foreign language university students’ writing", Computer Assisted Language Learning, 26(3), pp.234~257 (2013)
  41. Weir, C. J., "Continuity and innovation: The history of the CPE 913-2002, Studies in Language Testing 15", pp.2003, Cambridge University Press
  42. 이지연, "영작문 평가를 위한 채점자 훈련의 방향", 현대영어교육, 10(3), pp.217~241 (2009), 현대영어교육학회

이 간행물의 다른 논문 (12건)

The impact of a new challenge on teachers¡¯ professional development배연숙pp. 1-20Use and effects of the tutoring service at the English Writing Lab정영경pp. 21-44Scaffolding reading comprehension: A case study of a young Korean English language learner김숙희pp. 45-64An analysis of elementary students’ L2 meaning making in an EFL context김양희, Holbrook Mahnpp. 65-86영자신문제작 활동이 초등영어 학습자의 읽기-쓰기 능력 및 학습동기에 미치는 영향김유지, 김혜련pp. 87-106
원문 보기