LLM 기반 맞춤형 자동문항생성기의 활용가능성 탐색:Poe AI 챗봇빌더를 중심으로
Exploring the potential of LLM-based customized test item generators: Focusing on Poe AI as a chatbot builder
0
조회
0
다운로드
2
인용
0
참고문헌
키워드
초록
The study aims to enhance prompt literacy among pre-service teachers by developing an automated test item generator using Poe AI, a free chatbot builder platform based on Large Language Models (LLMs). An experiment was conducted with 84 pre-service teachers enrolled at a university of education in 2024. The experiment lasted two hours, with one hour dedicated to training and practice in chatbot creation, followed by one hour dedicated to creating an item generator and participating in a survey. The pre-service teachers successfully created curriculum-based item generators. However, due to time constraints, they could only focus on one to four item types. The survey revealed that 90.5% of the pre-service teachers had no prior experience in building a chatbot before the experiment. As a result, 65.5% found the process somewhat difficult, and 25.0% found writing prompts very difficult. Despite facing challenges, most feedback on the quality of the items generated by their programs was positive. Only 7.1% reported that the items were immediately usable, but 46.4% stated that the items could be used with minor modifications. These findings highlight the importance of providing prompt literacy training for pre-service teachers. In an open-ended questionnaire, the participants identified prompt literacy as a critical future core competency in the context of AI-human collaboration.
Abstract
The study aims to enhance prompt literacy among pre-service teachers by developing an automated test item generator using Poe AI, a free chatbot builder platform based on Large Language Models (LLMs). An experiment was conducted with 84 pre-service teachers enrolled at a university of education in 2024. The experiment lasted two hours, with one hour dedicated to training and practice in chatbot creation, followed by one hour dedicated to creating an item generator and participating in a survey. The pre-service teachers successfully created curriculum-based item generators. However, due to time constraints, they could only focus on one to four item types. The survey revealed that 90.5% of the pre-service teachers had no prior experience in building a chatbot before the experiment. As a result, 65.5% found the process somewhat difficult, and 25.0% found writing prompts very difficult. Despite facing challenges, most feedback on the quality of the items generated by their programs was positive. Only 7.1% reported that the items were immediately usable, but 46.4% stated that the items could be used with minor modifications. These findings highlight the importance of providing prompt literacy training for pre-service teachers. In an open-ended questionnaire, the participants identified prompt literacy as a critical future core competency in the context of AI-human collaboration.
참고문헌 (31)
- 신동광, "AI 도구를 활용한 중등교사 영어 출제 연수 사례 연구 : ChatGPT를 중심으로", Language Research, 59(1), pp.21~42 (2023)[DOI]
- 신동광, "생성형 AI를 활용한 영어 스토리북 제작이초등학생의 학습동기에 미치는 영향", 영어교과교육, 22(4), pp.177~196 (2023)[DOI]
- 황요한, "PROMPT 리터러시 시대의 도래에 관한 고찰 : 영어교육을 위한 ChatGPT와 DALL-E의 활용을 중심으로", 영어교과교육, 22(2), pp.263~288 (2023)[DOI]
- Attali, Y., "The interactive reading task : Transformer-based automatic item generation", Frontiers in Artificial Intelligence, 5, pp.903077~ (2022)[DOI]
- Arendasy, M., "Using automatic item generation to meet the increasing item demands of high-stakes assessment", Learning and Individual Differences, 22, pp.112~117 (2012)[DOI]
- "genQue" (2023)[URL]
- Atapattu, T., "Database and expert systems applications: DEXA 2012" (2012), Springer
- Attali, Y., "Automatic item generation unleashed: an evaluation of a large-scale deployment of item models", Proceedings of the 19th International Conference of Artificial intelligence in education (2018), Springer
- Bormuth, J., "On a theory of achievement test items" (1969), University of Chicago Press
- Drasgow, F., "Educational measurement", pp.471~516 (2006), Praeger Publishers
- "Q-Craft" (2024)[URL]
- Geerlings, H., "Modeling rule-based item generation", Psychometrika, 76, pp.337~359 (2011)[DOI]
- Gierl, M. J., "Instructional topics in educational measurement(items)module : Using automated processes to generate test items", Educational Measurement : Issues and Practice, 32, pp.36~50 (2013)[DOI]
- Gierl, M. J., "Evaluating the psychometric characteristics of generated multiple-choice test items", Applied Measurement in Education, 29, pp.196~210 (2016)[DOI]
- Gierl, M. J., "Advanced methods in automatic item generation" (2021), Routledge
- Gierl, M. J., "Using automatic item generation to create multiple-choice test items", Medical Education, 46, pp.757~765 (2012)[DOI]
- Gilster. P., "Digital literacy" (1997), Wiley Computer Pub
- Kandlhofer, M., "AI k-12 education service", KI Kunstliche Intelligenz, 35(2), pp.125~126 (2021)[DOI]
- Khodeir, N. A., "Generating story problems via controlled parameters in a web-based intelligent tutoring system", International Journal of Information and Learning Technology, 35, pp.199~216 (2018)[DOI]
- Kong, S. C., "Evaluation of an artificial intelligence literacy course for university students with diverse study backgrounds", Computers & Education : Artificial Intelligence, 2, pp.100026~ (2021)[DOI]
- "ParaQG: A system for generating questions and answers from paragraphs" (2019)[URL]
- LaDuca, A., "Item modeling procedures for constructing content-equivalent multiple-choice questions", Medical Education, 20, pp.53~56 (1986)[DOI]
- "ChatGPT : My GPTs" (2023)[URL]
- Pugh, D., "Using cognitive models to develop quality multiple-choice questions", Medical Teacher, 38, pp.838~843 (2016)[DOI]
- Pugh, D., "Can automated item generation be used to develop high quality MCQs that assess application of knowledge", Research and Practice in Technology Enhanced Learning, 15, pp.1~13 (2020)[DOI]
- "Poe AI" (2023)[URL]
- Shin, D., "Can ChatGPT make reading comprehension testing items on par with human experts?", Language Learning & Technology, 27(3), pp.27~40 (2023)
- "Training optimus prime, M.D.: Generating medical certification items by fine tuning OpenAI’s gpt2 transformer model" (2019)[URL]
- Wang, Z., "QG net: A data driven question generation model for educational content", Proceedings of the Fith Annual ACM Conference on Learning at Scale, pp.1~10 (2018), ACM
- Zhou, Q., "Towards generating math word problems from equations and topics", Proceedings of the 12th international conference on natural language generation, pp.494~503 (2019), ACL
- "Neural question generation from text: a preliminary study" (2017)[URL]