A Study on a Deep Learning-Based Approach for Automated Scoring Solutions of Korean L1 Essays
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
The study utilized 401 data points categorized into upper, middle, and lower levels. Themodel development process included five stages: 1) data purification and preprocessing, 2)embedding, 3) data segmentation and shape conversion, 4) model training, and 5) modelperformance evaluation. The research employed BERT, a pre-trained model, to develop thegrading model. Performance evaluation of the trained model yielded an accuracy of 0.7377,precision of 0.6514, recall of 0.7377, and an F1 score of 0.6717. These results demonstrate arelatively high level of performance compared to previous studies on scoring Korean writtenand essay answers by L1 learners. As a result, the feasibility of developing an automaticscoring model using the BERT language model with small-scale data from a specific domain. Also, the model’s performance shows some generalizability, but further exploration isneeded for improvement. Limitations of the study include the use of writing samples gradedwithout considering grade levels, limited test data, difficulties in processing unregisteredtokens, and the inherent unexplainability of deep learning techniques. Further discussionsand considerations on how to more effectively utilize artificial intelligence in Koreanlanguage education should continue. Ongoing research on automatic grading is necessaryto provide accurate and detailed educational feedback to students. As automatic gradingresearch in the field of Korean language education advances, it is expected that high-qualityeducational interventions for students will become possible in the future.
Abstract
The study utilized 401 data points categorized into upper, middle, and lower levels. Themodel development process included five stages: 1) data purification and preprocessing, 2)embedding, 3) data segmentation and shape conversion, 4) model training, and 5) modelperformance evaluation. The research employed BERT, a pre-trained model, to develop thegrading model. Performance evaluation of the trained model yielded an accuracy of 0.7377,precision of 0.6514, recall of 0.7377, and an F1 score of 0.6717. These results demonstrate arelatively high level of performance compared to previous studies on scoring Korean writtenand essay answers by L1 learners. As a result, the feasibility of developing an automaticscoring model using the BERT language model with small-scale data from a specific domain. Also, the model’s performance shows some generalizability, but further exploration isneeded for improvement. Limitations of the study include the use of writing samples gradedwithout considering grade levels, limited test data, difficulties in processing unregisteredtokens, and the inherent unexplainability of deep learning techniques. Further discussionsand considerations on how to more effectively utilize artificial intelligence in Koreanlanguage education should continue. Ongoing research on automatic grading is necessaryto provide accurate and detailed educational feedback to students. As automatic gradingresearch in the field of Korean language education advances, it is expected that high-qualityeducational interventions for students will become possible in the future.
참고문헌 (53)
- Alikaniotis, D., "Automatic text scoring using neural networks", Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pp.715~725 (2016)
- Attali, Y., "Automated essay scoring with e-rater® V.2", Journal of Technology, Learning, and Assessment, 4, pp.1~30 (2006)
- Attali, Y., "Handbook of Automated Essay Evaluation: Current Applications and New Directions", pp.181~198 (2013), Routledge
- Burstein, J., "Handbook of Automated Essay Evaluation: Current Applications and New Directions", pp.55~67 (2013), Routledge
- Cho, H., "Automatic score range classification of Korean essays using deep learning-based Korean language models-The case of KoBERT & KoGPT2-", Journal of the International Network for Korean Language and Culture, 18, pp.217~241 (2021)
- Choi, S., "A study on the patterns of source text borrowing performance in the integrated writing of the secondary students focusing on citations and paraphrasing", Research on Writing, 38, pp.201~245 (2018)
- Choi, S., "A study on the development of Korean language and essay test items for Korea college scholastic ability test", Journal of CheongRam Korean Language Education, 91, pp.135~178 (2023)
- Devlin, J., "BERT: Pre-training of deep bidirectional transformers for language understanding", Proceedings of NAACL-HLT, 2019, pp.4171~4186 (2019)
- Dikli, S., "An overview of automated scoring of essays", Journal of Technology, Learning, and Assessment, 5 (2006)
- Faseeh, M., "Hybrid approach to automated essay scoring : Integrating deep learning embeddings with handcrafted linguistic features for improved accuracy", Mathematics, 12, pp.3416~ (2024)[DOI]
- "LLM-as-a-tutor in EFL writing education: Focusing on evaluation of student-LLM interaction" (2024)[URL]
- Han, S., "Empirical study on the loss functions of contrastive learning-based multi-scale BERT model for automated essay scoring", Journal of KIIT, 21, pp.51~63 (2023)
- Horbach, A., "Crosslingual content scoring in five languages using machine translation and multilingual transformer models", International Journal of Artificial Intelligence in Education, 34, pp.1294~1320 (2024)[DOI]
- Hussein, M. A., "Automated language essay scoring systems: A literature review", PeerJ Computer Science, 5, pp.e208~ (2019)
- Hussein, M. A., "A trait-based deep learning automated essay scoring system with adaptive feedback", International Journal of Advanced Computer Science and Applications, 11, pp.287~293 (2020)[DOI]
- Isaacs, T., "Key Concepts in Educational Assessment" (2013), SAGE Publications
- Kim, D., "Training set creation method for essay creativity score prediction models", Journal of KIIT, 21, pp.69~82 (2023)
- Kim, D., "Comparison of BERT-based model performance for automated Korean essay scoring", Proceedings of KIIT Conference, pp.435~439 (2023)
- Kim, S., "A Study on Automated Essay Evaluation Method for Argumentative Writing Task Using Deep Learning-Based Natural Language Processing Technique -Based on Model of Collaborative Scoring between Teacher Scorer and Machine Scorer", Doctor, pp.2022
- Kim, S., "Development of a Deep Learning Model for Automated Grading of Korean Essay", Master, pp.2024
- Kim, S., "Validation study of Q-matrix for argumentative writing assessment using cognitive diagnostic model", Research on Writing, 45, pp.181~222 (2020)
- Kumar, V., "Explainable automated essay scoring: Deep learning really has pedagogical value", Frontiers in Education, 5 (2020)[DOI]
- Landauer, T. K., "An introduction to latent semantic analysis", Discourse Processes, 25, pp.259~284 (1998)[DOI]
- Landauer, T. K., "Automatic essay assessment. Assessment in Education:Principles", Policy & Practice, 10, pp.295~308 (2003)[DOI]
- Lee, J., "A BERT-based automatic scoring model of Korean language learners’essay", Journal of Information Processing Systems, 18, pp.282~291 (2022)
- Lee, K., "Do it! Natural Language Processing using BERT and GPT" (2021), Easyspublishing
- Lee, S., "A Study on the Development of Korean Language Learning and Scoring Tools applying Artificial Intelligence", Grammar Education, 44, pp.29~54 (2022)
- Lee, Y., "Exploring a way to build an AI automated essay scoring model with insufficient data", Journal of Education & Culture(JOEC), 28, pp.25~42 (2022)
- Lee, Y., "The development study of an automated scoring program for Korean essays, PASTA-I", Journal of Educational Evaluation, 36, pp.711~730 (2023)
- McCulloch, W. S., "A logical calculus of the ideas immanent in nervous activity", The Bulletin of Mathematical Biophysics, 5, pp.115~133 (1943)
- Neural networks for automated essay grading, "Nguyen, H.", 1~11 (2016)
- Nitko, A. J., "Educational Assessment of Students" (2007), Pearson
- Page, E. B., "Automated essay scoring:A cross-disciplinary perspective", pp.43~54 (2003), Lawrence Erlbaum Associates Publishers
- Park, H., "Do it! Introduction to deep learning by honestly coding" (2019), EasysPublishing
- Substantializing Methods of Restricted and Extended Response Essay Assessment through Enforcing the Instruction-Assessment Alignment, "Park, H.", Korea Institute of Curriculum and Evaluation Research, ~, pp.Korea Institute of Curriculum and Evaluation Research Report RRE 2019-6 (2019)
- Substantializing Methods of Restricted and Extended Response Essay Assessment through Enforcing the Instruction-Assessment Alignment, "Park, H.", Korea Institute of Curriculum and Evaluation Research, ~, pp.Korea Institute of Curriculum and Evaluation Research Report RRE 2019-6 (2019)
- Park, J., "A study on the development of automated Korean essay scoring model using random forest algorithm", Brain, Digital, & Learning, 13, pp.131~146 (2023)
- Park, J., "Development of empirically-derived and descriptor-based diagnostic(EDD)checklist for the cognitive diagnostic assessment in argumentative writing", Research on Writing, 41, pp.131~160 (2019)
- A Study on the Development of Automated Scoring Method for Computer-Based Essay and Short Answer Question Type Assessment(I), "Park, J.", Korea Institute of Curriculum and Evaluation Research, ~, pp.Korea Institute of Curriculum and Evaluation Research Report RRE 2022-6 (2022)
- A Study on the Development of Automated Scoring Method for Computer-Based Essay and Short Answer Question Type Assessment (II), "Park, J.", Korea Institute of Curriculum and Evaluation Research, ~, pp.Korea Institute of Curriculum and Evaluation Research Report RRE 2023-7 (2023)
- Park, K., "Deep Learning Algorithm Exploration for Automated Korean essay Scoring", Journal of Educational Evaluation, 35(3), pp.465~488 (2022)[DOI]
- "The role of AI in English assessment" (2023)[URL]
- Powers, D. E., "Comparing the validity of automated and human essay scoring", ETS Research Report Series, 2000(2), pp.1~23 (2000)
- Ramesh, D., "An automated essay scoring systems : A systematic literature review", Artificial Intelligence Review, 55 (2022)[DOI]
- Ravichandiran, S., "Getting Started with Google BERT: Build and Train State-of-the-Art Natural Language Processing Models Using BERT" (2021), Packt Publishing
- Song, M., "Analysis on the Accuracy of Automated Scoring for Korean Large-scale Assessments", The Journal of Curriculum and Evaluation, 19(1), pp.255~274 (2016)
- Sun, C., "How to fine-tune bert for text classification?", Chinese computational linguistics: 18th China national conference, CCL 2019, pp.194~206 (2019), Springer International Publishing
- Taghipour, K., "A neural approach to automated essay scoring", Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp.1882~1891 (2016)
- Yu, D., "CLES-BERT : Contrastive learning-based BERT model for automated essay scoring", Journal of KIIT, 21, pp.31~43 (2023)
- Zhang, H., "Co-attention based neural network for source-dependent essay scoring", Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pp.399~409 (2018), Association for Computational Linguistics