인공지능 기반 자동평가의 현재와 미래: 서술형 문항에 관한 문헌 고찰과 그 너머
The Present and Future of AI-based Automated Evaluation: A Literature Review on Descriptive Assessment and Other Side
0
조회
0
다운로드
28
인용
0
참고문헌
키워드
초록
본 연구에서는 인공지능 기반의 서술형 문항 자동평가 시스템의 현재와 기술적 및 윤리적 도전을 살펴보고, 그 단기적인 활용 방안을 살펴보았다. 인공지능 서술형 문항 자동평가 시스템은 이미 광범위한 주제의 에세이 채점뿐만 아니라 정교한 과학적 개념을 묻는 문항에 대한 채점에도 사용될 만한 성능을 보이며 평가의 패러다임을 바꾸어가고 있다. 하지만 인공지능 서술형 자동평가 시스템은 (1) 자연어 처리의 기술적 한계, (2) 채점 신뢰도 문제, (3) 인간 사용자의 신뢰와 인위적 속임, (4) 고부담 시험에서의 윤리적 한계 등의 도전을 안고 있다. 이에 단기적으로는 인공지능 자동평가 시스템이 (1) 학습자에게 적응적인(adaptive) 학습 지원 도구로서, (2) 학습자에게 빠른 피드백을 제공하고, (3) 평가 준거의 빠른 분석을 가능케 하는 역할을 감당할 수 있다. 하지만 이러한 문제들이 기술적/윤리적으로 극복된다면, 인공지능 자동평가 시스템의 미래는 과학 등의 교과-특수적인 영역에서, 텍스트를 넘어서는 이미지 처리를 통해 (1) 시각적 표상에 대한 자동평가, (2) 실제 수행에 대한 자동평가, (3) 실시간 피드백과 과정중심 자동평가에 활용될 수 있을 것이다. 하지만 딥 러닝이 인공지능 기술의 급속한 발달을 주도하게 될 것을 누구도 예측하지 못하였듯이, 인공지능 자동평가 시스템의 미래적 가능성 역시 여전히 열려 있다고 하겠다.
Abstract
This study primarily reviews the present of AI(Artificial Intelligence)-based automated evaluation of descriptive assessment with its technological/ethical issues, to derive short-term strategies for utilization of it. AI-based automated evaluation system of descriptive assessment have been driving force of paradigm change of evaluation, being used for scoring usual essays and even scientific concept problems. However, it also encounters some challenges such as (1) technological limitation of natural language processing, (2) a matter of scoring reliability, (3) user trust and deliberate cheating, and (4) ethical issues with respect to high-stake exams. Thus in short-term range, AI-based automated evaluation system can be used as (1) adaptive supporting tool for students’ learning, (2) fast learning analytic tools for students’ response, and (3) analysis of evaluation criteria. Yet, once those problems are overcome in technological/ethical aspects, AI-based system could be used for automated evaluation of (1) visual representation of mental model, (2) authentic practice, and (3) real-time feedback and process-based evaluation, in the dimension of image processing over text processing, in subject-specific sense. Though, as nobody predicted deep learning algorithm to be mainstream of AI technology like nowadays, a future possibility of AI-based automated evaluation system would surely be divergent.
참고문헌 (54)
- 김유정, "교사의 과정 중심 평가 역량에 관한 사례 연구 -중⋅고등학교 과학 교사 사례를 중심으로-", 한국과학교육학회지, 39(6), pp.695~706 (2019), 한국과학교육학회[DOI]
- 김홍겸, "미래교육에서의 인간 교사와 인공지능 교사의 상호보완적 관계에 대한 소고", 교육문화연구, 24(6), pp.189~207 (2018), 교육연구소[DOI]
- 김지은, "중학생 영작문 실력 향상을 위한 자동 문법 채점 시스템 구축", 한국콘텐츠학회 논문지, 7(5), pp.36~46 (2007), 한국콘텐츠학회
- 노은희, "한국어 서답형 문항 자동채점 결과 비교 분석 - 국가수준 학업성취도 평가 국어, 사회, 과학 문항을 중심으로 -", 교육과정평가연구, 17(2), pp.99~122 (2014), 한국교육과정평가원
- 한국어 서답형 문항 자동채점 프로그램 개발 및 실용성 검증, "노은희", 한국교육과정평가원, ~, pp.연구보고 RRE 2014-6 (2014)
- 류병운, "자율주행자동차 사고의 법적 책임", 홍익법학, 19(1), pp.31~58 (2018), 법학연구소[DOI]
- 백영민, "포퓰리즘" (2017), 한울아카데미
- 성경희, "자동채점 프로그램을 적용한 대규모 평가 사회과 서답형 문항 채점 및 결과 분석", 시민교육연구, 48(2), pp.31~56 (2016), 한국사회과교육학회[DOI]
- 학업성취도 평가 서답형 문항 컴퓨터 채점화 방안 탐색, "성태제", 한국교육과정평가원, ~, pp.연구보고 RRE 2010-1 (2010)
- 송미영, "대규모 평가 서답형 문항 채점을 위한 문장 수준 자동채점 프로그램의 정확성 분석", 교육과정평가연구, 19(1), pp.255~274 (2016), 한국교육과정평가원
- 장병탁, "장교수의 딥러닝" (2017), 홍릉과학출판사
- 조일현, "학습분석학의 이해" (2019), 피와이메이트
- 하민수, "영어기반 컴퓨터자동채점모델과 기계번역을 활용한 서술형 한국어 응답 채점 - 자연선택개념평가 사례 -", 한국과학교육학회지, 36(3), pp.389~397 (2016), 한국과학교육학회[DOI]
- 하민수, "학습 도구로서의 서술형 평가 그리고 인공지능의 활용: WA3I 프로젝트 사례", 현장과학교육, 13(3), pp.271~282 (2019), 한국현장과학교육학회[DOI]
- Aleven, V., "Automated, unobtrusive, action-by-action assessment of self-regulation during learning with an intelligent tutoring system", Educational Psychologist, 45(4), pp.224~233 (2010)
- Beggrow, E. P., "Assessing scientific practices using machine-learning methods: How closely do they match clinical interview performance?.", Journal of Science Education and Technology, 23(1), pp.160~182 (2014)
- Clauser, B. E., "Development of automated scoring algorithms for complex performance assessments: A comparison of two approaches", Journal of Educational Measurement, 34(2), pp.141~161 (1997)
- Devlin, J., "Bert: Pre-training of deep bidirectional transformers for language understanding", arXiv preprint arXiv:1810.04805, ~ (2018)
- Dori, Y. J., "Assessing high school chemistry students’ modeling sub-skills in a computerized molecular modeling learning environment", Instructional Science, 40(1), pp.69~91 (2012)
- Gerard, L. F., "Automated guidance for student inquiry", Journal of Educational Psychology, 108(1), pp.60~81 (2016)
- Ha, M., "The impact of misspelled words on automated computer scoring : A case study of scientific explanations", Journal of Science Education and Technology, 25(3), pp.358~374 (2016)
- Ha, M., "Predicting the accuracy of computer scoring of text: Probabilistic, multi-model, and semantic similarity approaches", Proceedings of the National Association for Research in Science Teaching (2016)
- Ha, M., "Applying computerized-scoring models of written biological explanations across courses and colleges: Prospects and limitations", CBE-Life Sciences Education, 10(4), pp.379~393 (2011)
- Haudek, K. C., "What are they thinking? Automated analysis of student writing about acid–base chemistry in introductory biology", CBE-Life Sciences Education, 11(3), pp.283~293 (2012)
- Ibabe, I., "Online self-assessment with feedback and metacognitive knowledge", Higher Education, 59(2), pp.243~258 (2010)
- Automatic assessment of complex task performance in games and simulations, "Iseli, M. R.", National Center for Research on Evaluation, Standards, Student Testing, Center for Studies in Education, UCLA, 775~, pp.CRESST Research Report No. 775 (2010)
- Kaplan, J. J., "Using lexical analysis software to assess student writing in statistics", Technology Innovations in Statistics Education, 8(1) (2014)
- Kukich, K., "The Debate on Automated Essay Grading", pp.27~31 (2000), IEEE Intelligent systems
- Lee, H. S., "Automated text scoring and real‐time adjustable feedback: Supporting revision of scientific arguments involving uncertainty", Science Education, 103(3), pp.590~622 (2019)
- Liu, O. L., "Automated scoring of constructed‐response science items: Prospects and obstacles", Educational Measurement: Issues and Practice, 33(2), pp.19~28 (2014)
- Liu, O. L., "Validation of automated scoring of science assessments", Journal of Research in Science Teaching, 53(2), pp.215~233 (2016)
- Lochbaum, K. E., "Detection of gaming in automated scoring of essays with the IEA", National Council on Measurement in Education Conference (NCME) (2013)
- Magnusson, S. J., "Dynamic science assessment: A new approach for investigating conceptual change", The Journal of the Learning Sciences, 6(1), pp.91~142 (1997)
- Mao, L., "Validation of automated scoring for a formative assessment that employs scientific argumentation", Educational Assessment, 23(2), pp.121~138 (2018)
- McMillan, J. H., "Student self-assessment: The key to stronger student motivation and higher achievement", Educational Horizons, 87(1), pp.40~49 (2008)
- Mikolov, T., "Efficient estimation of word representations in vector space", arXiv preprint arXiv:1301.3781, ~ (2013)
- Moharreri, K., "EvoGrader: An online formative assessment tool for automatically evaluating written evolutionary explanations", Evolution: Education and Outreach, 7(1), pp.15~ (2014)
- Nehm, R. H., "Item feature effects in evolution assessment", Journal of Research in Science Teaching, 48(3), pp.237~256 (2011)
- Nehm, R. H., "Human vs. computer diagnosis of students’ natural selection knowledge: Testing the efficacy of text analytic software", Journal of Science Education and Technology, 21(1), pp.56~73 (2012)
- Nehm, R. H., "Reasoning about natural selection : Diagnosing contextual competency using the ACORNS instrument", The American Biology Teacher, 74(2), pp.92~98 (2012)
- Nehm, R. H., "Transforming biology assessment with machine learning: Automated scoring of written evolutionary explanations", Journal of Science Education and Technology, 21(1), pp.183~196 (2012)
- Opfer, J. E., "Cognitive foundations for science assessment design: Knowing what students know about evolution", Journal of Research in Science Teaching, 49(6), pp.744~777 (2012)
- Page, E. B., "The imminence of... grading essays by computer", The Phi Delta Kappan, 47(5), pp.238~243 (1966)
- Powers, D. E., "Stumping e-rater: Challenging the validity of automated essay scoring", Computers in Human Behavior, 18(2), pp.103~134 (2002)
- Raczynski, K., "Appraising the scoring performance of automated essay scoring systems-Some additional considerations: Which essays? Which human raters? Which scores?", Applied Measurement in Education, 31(3), pp.233~240 (2018)
- Radford, A., "Language models are unsupervised multitask learners", OpenAI Blog, 1(8) (2019)
- Rudner, L. M., ". An evaluation of IntelliMetric™ essay scoring system", The Journal of Technology, Learning and Assessment, 4(4) (2006)
- Sripathi, K. N., "Mixed Student Ideas about Mechanisms of Human Weight Loss", CBE-Life Sciences Education, 18(ar37), pp.1~17 (2019)
- Tansomboon, C., "Designing automated guidance to promote productive revision of science explanations", International Journal of Artificial Intelligence in Education, 27(4), pp.729~757 (2017)
- Tasker, R., "Visualization: Theory and practice in science education", pp.103~131 (2008), Springer
- Weston, M., "Examining the impact of question surface features on students’ answers to constructed-response questions on photosynthesis", CBE-Life Sciences Education, 14(ar19), pp.1~12 (2015)
- Yang, Y., "A review of strategies for validating computer-automated scoring", Applied Measurement in Education, 15(4), pp.391~412 (2002)
- Zhu, M., "Investigating the impact of automated feedback on students’ scientific argumentation", International Journal of Science Education, 39(12), pp.1648~1668 (2017)
- Zhu, M., "The effect of automated feedback on revision behavior and learning gains in formative assessment of scientific argument writing", Computers & Education, 143, pp.103668~ (2020)