대학 글쓰기 평가에서 인공지능 도구의 활용 가능성 탐색: 인간과 생성형 AI 간 평가 기준 비교
Exploring the Potential of AI Tools in University Writing Assessment: Comparing Evaluation Criteria between Humans and Generative AI
3
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
본 연구는 Learning with AI 관점에서 출발하여, 인공지능이 생성한 글쓰기 평가 기준의 교육적 활용 가능성을 탐색하고자 하 였다. 구체적으로, 인공지능이 생성한 평가 기준과 인간이 개발한 기준 사이의 공통점과 차이점을 체계적으로 분석하고자 하 였다. 이를 위한 연구 문제는 1) 인공지능 도구가 생성한 글쓰기 평가 기준은 어떤 특성을 가지는가? 2) 인간과 인공지능 도구가 생성한 글쓰기 평가 기준은 서로 어떠한 공통점과 차이점을 갖는가?로 설정하였다. GPT와 Claude를 대표적인 인공지능 도구로 선정하여 대학생 글쓰기 평가 기준을 생성하게 한 후, 그 결과물을 인간이 만든 글쓰기 평가 기준과 대조하였다. 연구 결과, 인 간과 인공지능 도구 모두 글의 내용과 관련한 평가 범주에 가장 높은 중요도를 부여한다는 공통점을 보였다. 그러나, 인간은 내 용, 조직, 어법 등 세 개의 주요 범주로 평가하였으나, 인공지능 도구들은 형식 및 인용, 독창적(비판적) 사고, 전체적 인상 등의 추가 범주를 포함하여 평가 기준을 제시하였다. 전반적으로 인간은 각 평가 범주 내에서 상세한 항목을 포함하는 반면, 인공지 능 도구들은 간결하게 항목을 설정하였다. 특히, 인공지능 도구가 영어를 기반으로 개발되었기 때문에 발생하는 언어적 차이점 과 각 항목별 배점 체계와 관련한 차이점이 발견되었다. 이를 통해, 인간과 인공지능의 협력적 평가 모델 개발에 대한 중요한 시 사점을 제시하였으며, 향후 교육평가 장면에서 인공지능의 보완적 도구로서의 역할을 탐색하였다.
Abstract
This study, from the perspective of Learning with AI, aimed to explore the educational applicability of writing evaluation criteria generated by artificial intelligence. Specifically, it sought to systematically analyze the similarities and differences between AI generated criteria and those developed by humans. The research questions for this study were set as follows: 1) What characteristics do the writing evaluation criteria generated by AI tools have? 2) What similarities and differences exist between the writing evalu ation criteria generated by humans and AI tools? GPT and Claude were selected as representative AI tools, and they were tasked with generating writing evaluation criteria for undergraduate students. These AI-generated criteria were then compared with humancreated criteria. The results showed a commonality: Both humans and AI-tools placed the highest importance on categories relatedto content. However, while humans evaluated based on three main categories – content, organization, and language usage – the AItools included additional categories such as format and citations, original thinking, and overall impression. In general, human tendedto include more detailed items within each evaluation category, while AI tools presented more concise items. Notably, differenceswere observed in language-related aspects and scoring systems, which were influenced by the AI tools being developed based onEnglish. This study offers important insights into the development of collaborative evaluation models between humans and AI, andit explores the potential role of AI as a complementary tool in educational assessment in the future.
참고문헌 (26)
- W. Holmes, "Artificial intelligence in education: Promise and implications for teaching and learning" (2019), Center for Curriculum Redesign
- O. Zawacki-Richter, "Systematic review of research on artificial intelligence applications in higher education – where are the educators?", International Journal of Educational Technology in Higher Education, 16(1), pp.39~66 (2019)
- Artificial intelligence and edutech in school education, "S. Hong", Korea Institute for Curriculum and Evaluation, ~ (2020)
- S. Park, "Exploring the possibility of science-inquiry competence assessment by ChatGPT-4 : Comparisons with human evaluators", Korean Journal of Educational Research, 61(4), pp.299~332 (2023)
- S. Park, "A study on exploring the potential of ChatGPT in writing skills assessment : Focusing on essay writing", Korean Journal of Educational Research, 62(5), pp.219~248 (2024)
- J. W. Sung, "Exploring the feasibility of automatic scoring of written test using ChatGPT : Focusing on the world geography written test", Journal of the Association of Korean Geographers, 12(3), pp.415~432 (2023)
- W. J. Yoon, "A study on the performance of ChatGPT in evaluating coverletters and its potential to robotic process automation", Journal of Organization and Management, 47(4), pp.27~51 (2023)
- E. H. Ham, "Characteristics of GPT-4 automated scoring of scientific inquiry competency", The Journal of Educational Information and Media, 30(3), pp.713~742 (2024)
- R. H. Nehm, "Transforming biology assessment with machine learning : automated scoring of written evolutionary explanations", Journal of Science Education and Technology, 21(1), pp.183~196 (2011)
- S. Oh, "Analysis of generative AI's mathematical problem-solving performance : Focusing on ChatGPT 4, Claude 3 Opus, and Gemini Advanced", The Mathematical Education, 63(3), pp.549~571 (2024)
- H. Kim, "Service design-focused comparative analysis of intelligent expert assistant(IEA)and taskoriented rule-based chatbot", Journal of Digital Contents Society, 25(6), pp.1443~1452 (2024)
- J. S. Kim, "The current status and future direction of communication competency subject according to changes in university education", Journal of Humanities, 35, pp.185~217 (2023)
- S. Y. Lee, "A study of the instructors’ perceptions on university writing and key competencies", Research on Writing, 20, pp.135~163 (2014)
- S. Kim, "A study on automated essay evaluation method for argumentative writing task using deep learning based natural language processing technique - Based on model of collaborative scoring between teacher scorer and machine scorer", Ph. D., pp.2022
- Substanitializing methods of restricted and extended response essay assessment through enforcing the instruction-assessment alignment, "H. Y. Park", Korea Institute for Curriculum and Evaluation, ~ (2019)
- Y. S. Lee, "Currents and suggestions of scoring reliability and validity", Korea Institute for Curriculum and Evaluation, ~ (2013)
- Y. M. Park, "The method and procedure of evaluating writing ability", The Education of Korean Language, 99, pp.1~29 (1999)
- Development of a national writing skills assessment system in 2023, "B. G. Min", National Institute of Korean Language, ~ (2023)
- B. G. Min, "Development of a writing ability diagnosis system for university first-year students – The Seoul National University Essay Examination", The Korean Journal of Literacy Research, 13(6), pp.45~76 (2022)
- S. Park, "An exploratory study on developing the AI essay test tool based on ChatGPT : Focusing on the interaction with the engineer", Journal of Practical Engineering Education, 16(1), pp.21~31 (2024)
- C. K. Lo, "What is the impact of ChatGPT on education? A rapid review of the literature", Education Sciences, 13(4), pp.410~425 (2023)
- D. Lee, "Exploration on the feasibility of utilization and teacher perceptions of using Chat-GPT for student assessment in Science", Journal of the Korean Association for Science Education, 44(1), pp.119~130 (2024)
- "Human-level comprehension” Claude 3 unveiled: ChatGPT and Gemini on edge... Anthropic shaking up the generative AI landscape" (2024)[URL]
- "Generative AI, Encroaching on human creativity!(62) How far ahead is Anthropic’s Claude compared to ChatGPT-4? (part 2)" (2024)[URL]
- "Generative AI, Encroaching on human creativity!(61) How far ahead is Anthropic’s Claude compared to ChatGPT-4? (part 1)" (2024)[URL]
- S. Lee, "A study on the evaluation of expository writing skills of middle school students", Korean Literature & Language Education, 40, pp.71~99 (2022)