본문 바로가기
KERD
논문KCI 등재

문항 및 피드백 생성에서 규칙기반모형과 대규모 언어모형 비교

Comparison of Rule-Based Models and Large Language Models in Item and Feedback Generation

김성연|정진민
과학교육연구소 (Science Education Research Institute)|2024.12|과학교육연구지 (Journal of Science Education)|제48권 제3호|pp. 154-169|DOI 10.21796/jse.2024.48.3.154

0

조회

0

다운로드

0

인용

0

참고문헌

키워드

규칙기반모형대규모 언어모형문항 생성수와 연산피드백 생성feedback generationitem generationlarge language modelsnumber and operationrule-based models

초록

본 연구의 목적은 규칙 기반 모형과 대규모 언어모형을 사용한 문항 및 피드백 생성 방법을 정확성 측면에서 비교 분석하고, 실제 교육 현장에서의 적용 가능성을 탐색하는 것이다. 이를 위해 예시 자료로 2022 개정 교육과정의 중학교 수와 연산 영역에서 내용타당도를 확보한 7개의 성취 수준에 대해 문항과 피드백을 생성하였다. 주요 분석 결과는 다음과 같다. 첫째, 규칙 기반 모형에 따른 문항과 피드백 생성의 정확도는 100%였으나, 대규모 언어모형에 따른 문항과 피드백 생성의 정확도는 0%~100%로 다양하게 나타났다. 문항 생성에서 나타나는 오류로는 선택지와 관련하여 정답이 여러 개인 경우와 답이 없는 경우로 구분되었다. 둘째, 오류가 지속적으로 나타나는 제곱근의 제곱근을 묻는 문항의 경우에는 영어로 프롬프트를 작성하거나 대규모 데이터를 활용한 학습이 필요한 것으로 나타났다. 셋째, 효율적인 맞춤형 평가를 위해 규칙 기반으로 문항을 생성함으로써 평가의 정확성을 확보하고, 언어모형을 기반으로 학생 개개인의 정의적 특성, 진로, 학업 수준 등을 반영한 효율적인 피드백을 생성할 수 있는 것으로 나타났다. 마지막으로 본 연구의 제한점과 향후 연구 방향에 대해 논하였다.

Abstract

The purpose of this study is to compare and analyze the accuracy of item and feedback generation methods using rule-based models and large language models, and to explore their applicability in real educational settings. To accomplish this, items and feedback were generated for seven achievement levels validated for content in the middle school “Numbers and Operations” area of the 2022 revised curriculum. The main findings are as follows. First, the accuracy of item and feedback generation using rule-based models was 100%, whereas the accuracy of generation using large language models varied between 0% and 100%. Errors in item generation were categorized into cases where multiple correct answers were associated with choices and cases where no answer was provided. Second, for items repeatedly showing errors, such as those asking for the square root of a square root, it appeared necessary to write prompts in English or to train with large-scale data. Third, it was found that creating items based on rule-based models ensures the accuracy of assessments, while generating efficient feedback that reflects individual affective traits, career paths, and academic levels using large language models. Finally, the study discusses limitations and directions for future research.

참고문헌 (51)

  2. Attali, Y., "Effects of feedback elaboration and feedback timing during computer-based practice in mathematics problem solving", Computers & Education, 110, pp.154~169 (2017)
  3. Babitha, M. M., "Trends of Artificial Intelligence for online exams in education", International journal of Early Childhood special Education, 14(01), pp.2457~2463 (2022)
  4. Bandalos, D. L., "Measurement theory and applications for the social sciences" (2017), Guilford Publications
  5. Bejar, I. I., "Automatic item generation", pp.40~55 (2012), Routledge
  7. Butler, D. L., "Feedback and self-regulated learning : A theoretical synthesis", Review of educational research, 65(3), pp.245~281 (1995)
  8. Chen, X., "Application and theory gaps during the rise of artificial intelligence in education", Computers and Education : Artificial Intelligence, 1, pp.100002~ (2020)
  9. Choi, J., "Collective AI on the Foundation AI: The pathway of digital transformation of intelligence" (2024), CAFA Lab, Inc
  10. Choi, J., (2012), CAFA Lab, Inc
  11. Choi, J., (2012), CAFA Lab, Inc
  12. Choi, J., "CAFA model example book" (2022), Pubple
  13. Devlin, J., "BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding", Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pp.4171~4186 (2019)
  14. Embretson, S. E., "A cognitive design system approach to generating valid tests : Application to abstract reasoning", Psychological methods, 3(3), pp.380~ (1998)
  15. Embretson, S. E., "Understanding and quantifying cognitive complexity level in mathematical problem solving items", Psychology Science, 50(3), pp.328~ (2008)
  16. Embretson, S. E., "Automatic Item Generation: Theory and Practice" (2013), Routledge
  18. Gardner, J., "Artificial intelligence in educational assessment : ‘Breakthrough? Or buncombe and ballyhoo?’", Journal of Computer Assisted Learning, 37(5), pp.1207~1216 (2021)
  19. Gierl, M. J., "Using Automatic Item Generation to create multiple-choice test items", Medical Education, 46(8), pp.757~765 (2012)
  20. Gierl, M. J., "Automatic item generation: Theory and practice" (2013), Routledge
  21. Gierl, M., "The role of item models in automatic item generation", International Journal of Testing, 12(3), pp.273~298 (2012)
  22. Gierl, M., "Using Automated Processes to Generate Test Items And Their Associated Solutions and Rationales to Support Formative Feedback", IxD&A, 25, pp.9~20 (2015)
  24. Exploring the impact of large language models on educational feedback in classroom settings, "Harvard University.", Harvard Graduate School of Education, ~ (2023)
  25. Advances in AI and machine learning for education research. Computational Psychometrics: New Methodologies for a New Generation of Digital Learning and Assessment: With Examples in R and Python, "Huang, Y.", 195~208 (2021)
  26. Huff, K., "Automatic item generation", pp.102~117 (2012), Routledge
  27. Irvine, S. H., "Item generation for test development" (2002), Lawrence Erlbaum Associates
  28. Ji, Z., "Survey of hallucination in natural language generation", ACM Computing Surveys, 55(12), pp.1~38 (2023)
  29. Kasneci, E., "ChatGPT for good? On opportunities and challenges of large language models for education", Learning and individual differences, 103, pp.102274~ (2023)
  30. Kim, S., "Suggestions on the use of the CAFA system to promote teachers’ digital competencies", The Journal of the Korea Contents Association, 23(4), pp.475~493 (2023)
  31. Kulik, J. A., "Effectiveness of intelligent tutoring systems : A meta-analytic review", Review of Educational Research, 86(1), pp.42~78 (2016)
  32. Kwon, O., "Analyzing mathematical performances of ChatGPT : Focusing on the solution of national assessment of educational achievement and the college scholastic ability test", Communications of Mathematical Education, 37(2), pp.233~256 (2023)
  33. Laverghetta Jr, A., "Generating better items for cognitive assessments using large language models", Proceedings of the 18th workshop on innovative use of NLP for building educational applications (BEA 2023), pp.414~428 (2023)
  34. Lee, G., "Exploring the potential of ChatGPT in mathematics education : Analyzing mathematical performance and errors through a multimodal approach", Journal of the Korean School Mathematics Society, 27(3), pp.363~392 (2024)
  35. Lee, H., "A comparative analysis of the mathematical capabilities of ChatGPT 3. 5 and 4. 0-Focused on the 8th grade mathematics textbooks-", The Journal of the Korea Contents Association, 24(7), pp.436~449 (2024)
  37. Maynez, J., "On faithfulness and factuality in abstractive summarization", Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.1906~1919 (2020)
  38. Meissner, R., "LLM-generated competence-based e-assessment items for higher education mathematics: methodology and evaluation", Frontiers in Education, 9, pp.1427502~ (2024)
  39. "GPT-4 Technical Report" (2023)[URL]
  40. Radford, A., "Language models are unsupervised multitask learners", OpenAI blog, 1(8), pp.9~ (2019)
  42. Improvements in automated feedback tools enhance teacher-student communication and student satisfaction, "Stanford University.", Stanford University, ~ (2023)
  43. Shute, V. J., "Focus on formative feedback", Review of educational research, 78(1), pp.153~189 (2008)
  44. Tapalova, O., "Artificial intelligence in education : AIEd for personalised learning pathways", Electronic Journal of e-Learning, 20(5), pp.639~653 (2022)
  45. VanLehn, K., "The behavior of tutoring systems", International Journal of Artificial Intelligence in Education, 16(3), pp.227~265 (2006)
  46. Vaswani, A., "Attention is All You Need", Advances in Neural Information Processing Systems, 30, pp.5998~6008 (2017)
  47. von Davier, A. A., "Computational psychometrics: New methodologies for a new generation of digital learning and assessment: With examples in R and Python" (2022), Springer Nature
  48. Wei, J., "Chain-of-thought prompting elicits reasoning in large language models", Advances in neural information processing systems, 35, pp.24824~24837 (2022)
  49. Wiliam, D., "Embedded formative assessment" (2011), Solution tree press
  51. Zhou, K., "Learning to prompt for vision-language models", International Journal of Computer Vision, 130(9), pp.2337~2348 (2022)

관련 논문AI추천

거대 언어 모델을 활용한 일차방정식 문항의 오개념 기반 오답지 자동 생성신인섭, 유연주, 조상연한국수학교육학회 · 수학교육2025 · 제64권 제3호사회과 자동문항생성을 위한 대규모 언어모델의 활용 가능성 탐색이정우, 이현숙, 임상묵, 조혜원한국교육정보미디어학회 · 교육정보미디어연구2024 · 제30권 제3호AI 기반 논⋅서술형 평가 시스템의 국내⋅외 사례 및 향후 전망강태훈, 김동호, 박찬호, 이재화, 최상범교육문제연구소 · 교육연구2025 · 제94권 In-Context Learning을 통한 서·논술형 평가에서의 대형언어 모델과 교사 간 채점 및 피드백 정합성 향상 방안김강래, 이은아, 이주영뇌·AI기반교육연구소 · Brain, Digital, & Learning2025 · 제15권 제3호수학 구성형 답안의 자동 채점 연구에 관한 체계적 문헌 고찰김수훈, 하민수한국교육과정평가원 · 교육과정평가연구2024 · 제27권 제4호생성형 AI 기반 한국어 글쓰기 피드백의 질 평가 도구 개발 및 타당화 연구최숙기청람어문교육학회 · 청람어문교육2025 · 제103호

이 간행물의 다른 논문 (12건)

과학교육 전문가의 ChatGPT 활용 SSI 교육 개념틀 개발 과정 분석이지애, 박현주, 최성연, 이화선, 정진수pp. 131-143과학자의 실험과 과학 수업에서 실험 목적에 대한 초등 3학년생들의 인식 비교장명덕pp. 144-153초등학교 환경교육 관련 성취 기준 및 내용 요소의 분석박재근pp. 170-181환경·지속가능발전 교육을 위한 협업 네트워크 모델 탐색오현정, 김서현, 손연아, 박병열pp. 182-1972022 개정 수학과 교육과정에 따른 초등학교 1-2학년군 국정 교과용 도서의 덧셈과 뺄셈 단원 개발 과정에 대한 쟁점 연구이대현pp. 198-211
원문 보기