본문 바로가기
KERD
논문KCI 등재

수학 구성형 답안의 자동 채점 연구에 관한 체계적 문헌 고찰

A Systematic Literature Review on Automated Scoring of Mathematical Constructed Responses

하민수|김수훈
한국교육과정평가원 (Korea Institute for Curriculum and Evaluation)|2024.11|교육과정평가연구 (The Journal of Curriculum and Evaluation)|제27권 제4호|pp. 293-324|DOI https://doi.org/10.29221/jce.2024.27.4.293

1

조회

0

다운로드

0

인용

0

참고문헌

키워드

인공지능Artificial Intelligence서술형 평가자동 채점automated scoringdescriptive assessmentresponse type채점 방법scoring method수학 구성형 답안답안의 형태mathematical constructed response
주제분류교육학

초록

자동 채점은 최근 발달한 인공지능 기술을 학습자 평가에 적용함으로써 구성주의적 교육 평가의 실현에 도움을 줄 수 있는 연구 분야이다. 수학은 국가 교육과정의 기본적이면서도 필수적인 교과인 만큼 수학 문항에 대한 자동 채점 연구도 그 중요성이 상당하다. 그러나 타 교과의 자동 채점 연구와 달리 수학 교과를 대상으로 한 연구는 아직 시작 단계에 있다. 본 연구에서는 수학 교과에서 답안의 형태 및 입력 유형을 포함한 구성형 문항의 분류기준을 제안하고, 이를 바탕으로 체계적 문헌 고찰의 방법을 통해 SCOPUS에 등재된 15개 학술지에서 21편의 연구를 식별한 뒤 이를 분석하였다. 연구 결과, 수학 교과의 자동 채점 연구보고는 2010년대부터 조금씩 이루어지다가 2010년 후반부터 점차 증가하기 시작하였고, 중고등학교급을 대상으로 변화와 관계 및 수와 연산 영역의 채점 연구가 주를 이루었다. 인공지능 기술 발전에 따라 자동 채점 모델도 초기의 규칙 기반 및 통계 기반 모델에서 기계학습, 딥러닝, 대규모 언어모델로 변화했다. 단일 형태의 디지털 입력 답안을 분석한 초창기 자동 채점 연구는 점차 멀티모달 답안 또는 손글씨 답안을 자동 채점하는 방향으로 발전하고 있다. 이러한 연구 결과를 바탕으로 하여, 수학 교과의 구성형 답안 자동 채점 연구의 중요성과 필요성을 확인하고, 수행되어야 할 연구의 방향에 관한 제언을 하였다.

Abstract

Automated scoring is a research field that facilitates constructivist educational evaluation by applying recently developed artificial intelligence technology to learner evaluation. As mathematics is an essential subject of the K-12 curriculum, research on automated scoring of constructed mathematical responses is of great importance. However, unlike research on automated scoring in other subjects’ responses, studies for automated scoring of mathematical responses still require further exploration. In this study, we propose classification criteria for mathematical constructed responses including response types and input types, identified 21 studies from 15 academic journals registered in SCOPUS with systematic literature review, and investigated them in detail. As a result, Research on automated scoring of mathematical constructed responses has gradually increased in the 2010s, and mainly focusing on algebra and functions, number and operations at the secondary school level. With the advancement of artificial intelligence technology, automated scoring models have also evolved from early rule-based and statistical-based models to machine learning-, deep learning-, and large language models. Furthermore, early automated scoring studies which focused solely on single-modal and digital formatted answers are progressing toward automated scoring of multi-modal and handwritten answers. Based on these findings, We propose a framework that can classify research based on response types and input data formats. Finally, we conclude by emphasizing the importance and necessity of research in automated scoring of constructed mathematical responses, and by providing suggestions for future research directions.

참고문헌 (101)

  1. 김래영, "경기도 창의, 서술형 평가와 미국오하이오 주 평가 비교를 통한 중등 수학과 서술형 평가 체계 분석", 한국수학교육학회 학술발표논문집, 2013(1), pp.63~72 (2013)
  2. 김승주, "채점 자질 설계를 통한 지도 학습 기반 작문 자동 채점의 타당도 확보 방안 탐색", 청람어문교육, 69, pp.265~295 (2019)
  3. 박강윤, "순환신경망 장단기 기억(LSTM)을 이용한 자동 채점의 가능성 탐색", 교육과정평가연구, 24(4), pp.223~238 (2021)
  4. 박세진, "순환신경망을 적용한 초등학교 5학년 과학 서술형 평가 자동 채점시스템개발 및 활용 방안 모색", 교육평가연구, 33(2), pp.297~321 (2020)
  5. 컴퓨터 기반 서･논술형 평가를 위한 자동채점 방안 설계(Ⅰ), "박종임", 한국교육과정평가원, ~, pp.연구보고 RRE 2022-6 (2022)
  6. 신병철, "프롬프트 엔지니어링을 통한 GPT-4 모델의 수학 서술형 평가 자동 채점 탐색 : 순열과 조합을 중심으로", 수학교육, 63(2), pp.187~207 (2024)
  7. 이경건, "인공지능 기반 자동평가의 현재와 미래 : 서술형 문항에 관한 문헌 고찰과그 너머", 교육공학연구, 36(2), pp.353~382 (2020)
  8. 이상하, "국가수준 학업성취도 평가 서답형 문항에 대한 자동채점의 실용성분석", 교육과정평가연구, 18(1), pp.185~208 (2015)
  9. 이용상, "충분한 데이터 확보가 힘든 상황에서 인공지능 서･논술형 평가 채점모델 구축 방안", 교육문화연구, 28(5), pp.25~42 (2022)
  10. 이재봉, "합성곱 신경망(CNN)을 활용한 그래픽 답안 자동 채점 가능성 탐색", 새물리, 73(2), pp.138~149 (2023)
  11. 조희련, "딥러닝 기반 언어모델을 이용한 한국어 학습자 쓰기 평가의 자동 점수 구간 분류-KoBERT와 KoGPT2를 중심으로", 한국언어문화학, 18(1), pp.217~241 (2021)
  12. 최인용, "랜덤 포레스트 알고리즘을 활용한 수학 서술형 자동 채점", 수학교육, 63(2), pp.165~186 (2024)
  13. 최병홍, "Diagnostic Tree Model을 활용한 수학 서술형 문항 인지진단 평가 적용 연구", 수학교육학연구, 33(1), pp.1~25 (2023)
  14. Anderson, C. M., "Use of the structured descriptive assessment with typically developing children", Behavior Modification, 30(3), pp.352~378 (2006)
  15. Asakura, T., "Digitalizing educational workbooks and collecting handwritten answers for automatic scoring", 5th Workshop on Intelligent Textbooks (iTextbooks)@ AIED, pp.78~87 (2023)
  16. Baral, S., "Improving automated scoring of student open responses in mathematics", 14th International Conference on Educational Data Mining (EDM), pp.130~138 (2021)
  17. Baral, S., "Auto-scoring student responses with images in mathematics", 16th International Conference on Educational Data Mining (EDM), pp.362~369 (2023)
  18. Baral, S., "Enhancing auto-scoring of student open responses in the presence of mathematical terms and expressions", International Conference on Artificial Intelligence in Education, pp.685~690 (2022), Springer International Publishing
  19. Baxter, J. A., "Writing in mathematics : An alternative form of communication for academically low-achieving students", Learning Disabilities Research & Practice, 20(2), pp.119~135 (2005)
  20. Becker, J. P., "The open-ended approach : A new proposal for teaching mathematics" (2019), National Council of Teachers of Mathematics
  21. Bernal, M. E., "Revolutionizing eLearning assessments : The role of GPT in crafting dynamic content and feedback", Journal of Artificial Intelligence and Technology, 4(3), pp.188~199 (2024)
  22. Beseiso, M., "An empirical analysis of BERT embedding for automated essay scoring", International Journal of Advanced Computer Science and Applications, 11(10), pp.204~210 (2020)
  23. Blyth, B., "Assessment of e-mathematics with Maple", 9th Asian Technology Conference in Mathematics: ATCM, pp.143~152 (2004)
  24. Botelho, A., "Leveraging natural language processing to support automated assessment and feedback for student open responses in mathematics", Journal of Computer Assisted Learning, 39(3), pp.823~840 (2023)
  25. "Automated grading of handwritten numerical answers" (2017)[URL]
  26. Burstein, J., "Computer analysis of essays", NCME Symposium on Automated Scoring (1998)
  27. Campbell, C., "Sage handbook of research on classroom assessment", pp.71~84 (2012)
  28. Carlson, M., "Utilizing OpenAI’s GPT-4 for written feedback", TESOL Journal, 15(2), pp.e759~ (2023)
  29. Chaowicharart, E., "A step toward an automatic handwritten homework grading system for mathematics", Information Technology and Control, 52(1), pp.169~184 (2023)
  30. Dai, W., "Can large language models provide feedback to students? A case study on ChatGPT", 2023 IEEE International Conference on Advanced Learning Technologies (ICALT), pp.323~325 (2023), IEEE
  31. Devlin, J., "BERT : Pre-training of deep bidirectional transformers for language understanding", 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies, 1, pp.4171~4186 (2019)
  32. Erickson, J. A., "The automated grading of student open responses in mathematics", 10th International Conference on Learning Analytics & Knowledge, pp.615~624 (2020)
  33. Evens, H., "Categorizing pupils’ written answers to a mathematics test questions : ‘I know but I can’t explain’", Educational Research, 46(3), pp.269~282 (2004)
  34. Fernandez, N., "Automated scoring for reading comprehension via in-context BERT tuning", International Conference on Artificial Intelligence in Education, pp.691~697 (2022), Springer International Publishing
  35. Fife, J. H., "Automated scoring of mathematics tasks in the common core era:Enhancements to m-rater in support of CBAL™ mathematics and the common core assessments", ETS research report series, 2013(2), pp.i~35 (2013)
  36. The m-rater engine: Introduction to the automated scoring of mathematics items. Research Memorandum, "Fife, J. H.", ~, pp.ETS RM-17-02, 10-24 (2017)
  37. Fleiss, J. L., "The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability", Educational and Psychological Measurement, 33(3), pp.613~619 (1973)
  38. Gold, C., "Exploring the impact of handwriting recognition on the automated scoring of handwritten student answers", 202017th International Conference on Frontiers in Handwriting Recognition (ICFHR), pp.252~257 (2020), IEEE
  39. Hertzog, N. B., "Open-ended activities : Differentiation through learner responses", Gifted Child Quarterly, 42(4), pp.212~227 (1998)
  40. Hsu, S., "Attitudes surrounding an imperfect AI autograder", 2021 CHI Conference on Human Factors in Computing Systems, pp.1~15 (2021)
  41. Hwang, G. J., "Roles and research trends of artificial intelligence in mathematics education : A bibliometric mapping analysis and systematic review", Mathematics, 9(6), pp.1~19 (2021)
  42. Jescovitch, L. N., "Comparison of machine learning performance using analytic and holistic coding approaches across constructed response assessments aligned to a science learning progression", Journal of Science Education and Technology, 30(2), pp.150~167 (2021)
  43. Kadupitiya, J. C. S., "Automated assessment of multi-step answers for mathematical word problems", 2016 16th International Conference on Advances in ICT for Emerging Regions (ICTer), pp.66~71 (2016), IEEE
  44. Kojima, T., "Large language models are zero-shot reasoners", Advances in Neural Information Processing Systems, 35, pp.22199~22213 (2022)
  45. Kukreja, V., "Machine learning models for mathematical symbol recognition : A stem to stern literature analysis", Multimedia Tools and Applications, 81(20), pp.28651~28657 (2022)
  46. Lan, A. S., "Mathematical language processing: Automatic grading and feedback for open response mathematical questions", 2nd ACM Conference on Learning@ Scale, pp.167~176 (2015)
  47. Latif, E., "Fine-tuning ChatGPT for automatic scoring", Computers and Education : Artificial Intelligence, 6, pp.100210~ (2024)
  48. Leacock, C., "C-rater : Automated scoring of short-answer questions", Computers and the Humanities, 37, pp.389~405 (2003)
  49. Lee, J., "Automated assessment of student hand drawings in free-response items on the particulate nature of matter", Journal of Science Education and Technology, 32(4), pp.549~566 (2023)
  50. Liberati, A., "The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration", Annals of Internal Medicine, 151(4), pp.W~65 (2009)
  51. Liu, O. L., "Automated scoring of constructed-response science items : Prospects and obstacles", Educational Measurement : Issues and Practice, 33(2), pp.19~28 (2014)
  52. Liu, O. L., "Validation of automated scoring of science assessments", Journal of Research in Science Teaching, 53(2), pp.215~233 (2016)
  53. Constructed-response test questions: Why we use them; How we score them, "Livingston, S. A.", Educational Testing Service, ~, pp.R&D Connections. Number 11 (2009)
  54. McConney, A., "Science and mathematics achievement in Australia : The role of school socioeconomic composition in educational equity and effectiveness", International Journal of Science and Mathematics Education, 8(3), pp.429~452 (2010)
  55. Mendis, C., "Automatic assessment of student answers for geometric theorem proving questions", 2017 Moratuwa Engineering Research Conference (MERCon), pp.413~418 (2017), IEEE
  56. Mueller, P. A., "The pen is mightier than the keyboard : Advantages of longhand over laptop note taking", Psychological Science, 25(6), pp.1159~1168 (2014)
  57. Nakamoto, R., "Enhancing automated scoring of math self-explanation quality using LLM-generated datasets : A semi-supervised approach", Computers, 12(11), pp.217~ (2023)
  58. Namkung, J. M., "Comparing and validating four methods for scoring mathematics writing", Reading & Writing Quarterly, 36(2), pp.157~175 (2020)
  59. Nguyen, M. L., "Web-based mathematics testing with automatic assessment", PRICAI 2012: Trends in Artificial Intelligence: 12th Pacific Rim International Conference on Artificial Intelligence, 12, pp.347~358 (2012), Springer
  60. Noutsara, S., "Mistakes in mathematics problems solving based on newman’s error analysis on set materials", Journal La Edusci, 2(1), pp.20~27 (2021)
  61. Olenev, A. A., "Using the Maple computer algebra system to study mathematical induction", Journal of Physics: Conference Series, 1691(1), pp.012102~ (2020), IOP Publishing
  62. Organisciak, P., "Beyond semantic distance : Automated scoring of divergent thinking greatly improves with large language models", Thinking Skills and Creativity, 49, pp.101356~ (2023)
  63. Othman, N. L. I., "Computational technique for stepwise quantitative assessment of equation correctness", AIP Conference Proceedings, 1830(1), pp.020033~ (2017), AIP Publishing
  64. Othman, N. L. I., "Accurateness evaluation of an IR-based marking engine for mathematics assessment", 2010 International Conference on Information Retrieval & Knowledge Management (CAMP), pp.18~23 (2010), IEEE
  65. Page, E. B., "The imminence of... grading essays by computer", The Phi Delta Kappan, 47(5), pp.238~243 (1966)
  66. Page, E. B., "Computer grading of student prose, using modern concepts and software", The Journal of Experimental Educational, 62(2), pp.127~142 (1994)
  67. Page, M. J., "The PRISMA 2020 statement: An updated guideline for reporting systematic reviews", bmj, 372 (2020)
  68. Pankiewicz, M., "Large Language Models (GPT) for automating feedback on programming assignment", International Conference on Computers in Education, pp.68~77 (2023)
  69. Pejić, A., "Math proficiency prediction in computer-based international large-scale assessments using a multi-class machine learning model", 2021 IEEE 19th International Symposium on Intelligent Systems and Informatics (SISY), pp.49~54 (2021), IEEE
  70. Popham, W. J., "What's wrong-and what's right-with rubrics", Educational Leadership, 55(2), pp.72~75 (1997)
  71. Powell, S. R., "A synthesis of mathematics writing : Assessments, interventions, and surveys", Journal of Writing Research, 8(3), pp.493~530 (2017)
  72. Radford, A., "Learning transferable visual models from natural language supervision", International Conference on Machine Learning, pp.8748~8763 (2021), PMLR
  73. "Improving language understanding by generative pre-training" (2018)[URL]
  74. "Enhancing mathematical capabilities through ChatGPT and similar generative artificial intelligence: Roles and challenges in solving mathematical problems" (2023)[URL]
  75. Reimers, N., "Sentence-BERT: Sentence embeddings using siamese BERT-networks", 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp.3982~3992 (2019)
  76. Rivera-Bergollo, R., "Leveraging auxiliary data from similar problems to improve automatic open response scoring", 15th International Conference for Educational Data Mining, pp.679~683 (2022)
  77. Ryan, S. A., "Drawing for assessing learning outcomes in chemistry", Journal of Chemical Education, 96(9), pp.1813~1820 (2019)
  78. Rønning, F., "Influence of computer-aided assessment on ways of working with mathematics", Teaching Mathematics and Its Applications : International Journal of the IMA, 36(2), pp.94~107 (2017)
  79. Saarela, M., "Predicting math performance from raw large-scale educational assessments data: A machine learning approach", 33rd International Conference on Machine Learning, MLR Workshop and Conference Proceedings, pp.1~8 (2016), JMLR
  80. Santos, L., "The second handbook of research on the Psychology of Mathematics Education", pp.151~185 (2016), Brill
  81. Schneider, J., "Towards trustworthy autograding of short, multi-lingual, multi-type answers", International Journal of Artificial Intelligence in Education, 33(1), pp.88~118 (2023)
  82. Shaikh, E., "Automated grading for handwritten answer sheets using convolutional neural networks", 2019 2nd International Conference on New Trends in Computing Sciences (ICTCS), pp.1~6 (2019), IEEE
  83. Singley, M. K., "Validation and extension of the mathematical expression response type: Applications of schema theory to automatic scoring and item generation in mathematics", ETS Research Report Series, 1997(2), pp.i~43 (1997)
  84. Smolinsky, L., "Computer-based and paper-and-pencil tests : A study in calculus for STEM majors", Journal of Educational Computing Research, 58(7), pp.1256~1278 (2020)
  85. Taylor, C. S., "An investigation of scoring methods for mathematics performance-based asssessments", Educational Assessment, 5(3), pp.195~224 (1998)
  86. Trinh, T. H., "Solving olympiad geometry without human demonstrations", Nature, 625(7995), pp.476~482 (2024)
  87. Tvarožek, J., "Towards computerized adaptive assessment based on structured tasks", 5th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, pp.224~234 (2008)
  88. van Gerven, M., "Computational foundations of natural intelligence", Frontiers in Computational Neuroscience, 11, pp.112~ (2017)
  89. Vaswani, A., "Attention is all you need", Advances in Neural Information Processing Systems, 30 (2017)
  90. Wardat, Y., "ChatGPT: A revolutionary tool for teaching and learning mathematics", Eurasia Journal of Mathematics, Science and Technology Education, 19(7), pp.em2286~ (2023)
  91. Wei, J., "Chain-of-thought prompting elicits reasoning in large language models", Advances in Neural Information Processing Systems, 35, pp.24824~24837 (2022)
  92. Wijesinghe, D. B., "Automatic assessment of student answers consisting of venn and euler diagrams", 2017IEEE 17th International Conference on Advanced Learning Technologies (ICALT), pp.243~247 (2017), IEEE
  93. Wijeweera, B., "Automatic assessment of student answers for geometric construction questions", 2017 IEEE 17th International Conference on Advanced Learning Technologies (ICALT), pp.238~242 (2017), IEEE
  94. Williamson, D. M., "A framework for evaluation and use of automated scoring", Educational Measurement : Issues and Practice, 31(1), pp.2~13 (2012)
  95. Yang, C. W., "A HO-IRT based diagnostic assessment system with constructed response items", Turkish Online Journal of Educational Technology, 10(4), pp.46~51 (2011)
  96. Yao, S., "Tree of thoughts: Deliberate problem solving with large language models", 37th Conference on Neural Information Processing Systems (NeurIPS), pp.11809~11822 (2024)
  97. Yuhana, U. L., "Automatic assessment of answers to mathematics stories question based on tree matching and random forest", International Journal of Intelligent Engineering & Systems, 15(2), pp.200~212 (2022)
  98. Zhai, X., "A meta-analysis of machine learning-based science assessments : Factors impacting machine-human score agreements", Journal of Science Education and Technology, 30, pp.361~379 (2021)
  99. Zhai, X., "Applying machine learning in science assessment : A systematic review", Studies in Science Education, 56(1), pp.111~151 (2020)
  100. Zheng, K., "Minif2f: A cross-system benchmark for formal olympiad-level mathematics", 10th International Conference on Learning Representations (ICLR) (2021)
  101. Zhu, M., "Investigating the impact of automated feedback on students’ scientific argumentation", International Journal of Science Education, 39(12), pp.1648~1668 (2017)

관련 논문AI추천

랜덤 포레스트 알고리즘을 활용한 수학 서술형 자동 채점김화경, 송민호, 정인우, 최인용한국수학교육학회 · 수학교육2024 · 제63권 제2호AI 기반 수학 교수·학습에 대한 체계적 문헌 고찰: AI의 역할과 교사의 역할을 중심으로권오남, 윤정은한국수학교육학회 · 수학교육2024 · 제63권 제3호인공지능 기반 자동평가의 현재와 미래: 서술형 문항에 관한 문헌 고찰과 그 너머이경건, 하민수한국교육공학회 · 교육공학연구2020 · 제36권 제2호대규모 검사에서 구성형 수학문항을활용하기 위한 컴퓨터 채점의 타당화 분석김명화, 최대호한국교육개발원 · 한국교육2003 · 제30권 제3호AI 기반 논⋅서술형 평가 시스템의 국내⋅외 사례 및 향후 전망강태훈, 김동호, 박찬호, 이재화, 최상범교육문제연구소 · 교육연구2025 · 제94권 수학교육에서 인공지능 활용 가능성박만구한국수학교육학회 · 수학교육 논문집2020 · 제34권 제4호

이 간행물의 다른 논문 (12건)

사고의 통합을 위한 교과 교육과정 구성 방안 고찰: 탐구 과정을 중심으로윤지영pp. 1-24고교학점제 도입에 따른 최소 성취수준 보장 지도 지원 방안이광상pp. 25-48학습자 주도성 함양을 위한 학교 교육과정 설계･운영에서의 학생 참여 방안 탐색이수정, 이승미pp. 49-77CIPP 평가모형을 활용한 고교학점제 기반 특성화고 선택중심 교육과정 평가지표 개발 연구임해경pp. 79-103해외 저학년 음악과 교육과정 분석을 통한 2022 개정 즐거운 생활 고찰 및 향후 과제김아현, 송정주pp. 105-127
원문 보기