인공지능 시대의 작문 평가를 위한 ChatGPT 활용 방안 연구
Exploring the Utilization of ChatGPT for Composition Assessment in the Era of Artificial Intelligence
0
조회
0
다운로드
23
인용
0
참고문헌
키워드
초록
본 연구에서는 ChatGPT의 작문 교육 및 평가에서의 활용 가능성과 한 계를 중심으로 실제 학생글에 대한 채점과 피드백 수행을 통해 성능 수 준을 점검하고 작문 평가 분야에서의 ChatGPT와 같은 AI 모델 활용 방 안을 탐색하고자 하였다. 본 연구의 결과를 정리하면 다음과 같다. 첫째, ChatGPT의 자동 채점 성능과 가능성을 탐색한 결과 학생글에 대한 분석 과 신속한 채점이 가능하며, 전이 학습과 제로샷 러닝으로도 채점 루브릭 을 자동 생성하고 채점할 수 있는 기능이 확인되었다. 둘째, 고1 학생의 논증적 글쓰기를 대상으로, 국어 교사와 ChatGPT의 채점 결과를 비교한 결과, ChatGPT는 국어 교사 채점 결과와 2-3점의 근소한 차이를 보였으 나 채점 변동성 문제와 후속적인 정확성 검증을 거칠 필요가 있음이 확 인되었다. 이를통해작문채점의보조수단으로일부활용할수있는가 능성이 확인되었다. 셋째, ChatGPT의 작문 피드백 도구로의 성능과 활용 가능성을 평가한 결과, '즉시성'의 장점과 작문 과제 맥락에 폭넓게 적용 가능한 ‘확장성’, 작문 과정별 피드백 제공의 가능성이 확인되었다. 넷째, 소량의 사전 학습 데이터로 작문 피드백 생성이 가능하나 피드백 생성의 변동성 문제를 해소하고 효과적인 피드백 생성을 위한 프롬프트 엔지니 어링의 중요성이 확인되었다.
Abstract
In this study, we investigate the potential and limitations of employing ChatGPT for writing education and assessment, focusing on evaluating its performance through grading and feedback on actual student compositions. The research aims to explore the application of AI models like ChatGPT in the field of composition assessment. The findings of this study are summarized as follows: Firstly, When comparing evaluations of argumentative writing by first-year high school students between a Korean language teacher and ChatGPT, slight differences were observed, highlighting issues with grading variance and accuracy. Secondly, vThirdly, evaluating the performance and applicability of ChatGPT as a writing feedback tool demonstrates the advantages of "immediacy," wide applicability in various writing contexts through "scalability," and the potential for providing feedback across different stages of the writing process. Fourthly, while generating writing feedback is achievable with a small amount of pre-training data, the study highlights the significance of prompt engineering to mitigate variability issues in feedback generation and ensure effective feedback provision.
참고문헌 (35)
- 강동훈, "챗지피티(ChatGPT)의 등장과 국어교육의 대응", 국어문학, 82(82), pp.469~496 (2023), 국어문학회
- 교육부, "2022 국어과 교육과정", 교육부, ~ (2022)
- 박성미, "법률영역에서 GPT-4 활용 가능성과 시사점 - 법학적성시험(LEET) 중심으로 -", 경제규제와 법, 16(1), pp.7~28 (2023), 법학연구소[DOI]
- 장성민, "챗GPT가 바꾸어 놓은 작문교육의 미래 - 인공지능 시대, 작문교육의 대응을 중심으로 -", 작문연구, (56), pp.7~34 (2023), 한국작문학회[DOI]
- Adamopoulou, E., "Chatbots: History, technology, and applications", Machine Learning with Applications, 2 (2020)[DOI]
- Barrot, J. S., "Using ChatGPT for second language writing : Pitfalls and potentials", Assessing Writing, 57, pp.100745~ (2023)[DOI]
- Katz, Daniel, "GPT-4 Passes the Bar Exam", SSRN Electronic Journal (2023)[DOI]
- Caines, Andrew, "On the application of Large Language Models for language teaching and assessment technology", arXiv - CS - Machine Learning, ~ (2023)
- Choi, J. H., "ChatGPT Goes to Law School", SSRN, ~ (2023)
- Conijn, R., "The Effects of Explanations in Automated Essay Scoring Systems on Student Trust and Motivation", Journal of Learning Analytics, 10(1), pp.37~53 (2023)
- Dai, W., "Can Large Language Models Provide Feedback to Students?", A Case Study on ChatGPT, ~ (2023)
- Debby R. E. Cotton, "Chatting and cheating: Ensuring academic integrity in the era of ChatGPT", Innovations in Education and Teaching International (2023)[DOI]
- Dergaa, I., "From human writing to artificial intelligence generated text : examining the prospects and potential threats of ChatGPT in academic writing", Biology of sport, 40(2), pp.615~622 (2023)[DOI]
- Y. Elazar, "Measuring and Improving Consistency in Pretrained Language Models"", arXiv: 2102.01017, ~ (2021)
- Elkins, K., "Can GPT-3 pass a writer’s Turing Test", Journal of Cultural Analytics, 2371, pp.4549~ (2020)
- Floridi, Luciano, "AI as Agency Without Intelligence : on ChatGPT, Large Language Models, and Other Generative Models", Philosophy and Technology, 36(1), pp.1~7 (2023)
- Gilson, A., "How Does ChatGPT Perform on the United States Medical Licensing Examination? The Implications of Large Language Models for Medical Education and Knowledge Assessment", JMIR medical education, 9, pp.e45312~ (2023)[DOI]
- Godwin-Jones, R., "Partnering with AI : Intelligent writing assistance and instructed language learning", Language Learning & Technology, 26(2), pp.5~24 (2022)[DOI]
- Hackl, V., "Is GPT-4 a reliable rater? Evaluating Consistency in GPT-4 Text Ratings, Computation and Language", arXiv:2308.02575, 1~14 (2023)
- Hutson M., "Robo-writers : The rise and risks of language generating AI", Nature, 591(7848), pp.22~25 (2021)[DOI]
- Jeon, J., "Large language models in education: a focus on the complementary relationship between human teachers and ChatGPT", Education and Information Technologies (2023)[DOI]
- Kung TH, "Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models", PLOS Digit Health, 2(2), pp.e0000198~ (2023)[DOI]
- Duri Long, "What is AI Literacy? Competencies and Design Considerations", Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). Association for Computing Machinery, pp.1~16 (2020)
- McGarrell, H., "Motivating revision of drafts through formative feedback", ELT Journal, 61(3), pp.228~236 (2007)[DOI]
- Mizumoto, Atsushi, "Exploring the Potential of Using an Ai Language Model for Automated Essay Scoring", ~
- O'Connor, S., "Open Artificial Intelligence Platforms in Nursing Education: Tools for Academic Progress or Abuse?", Nurse Education in Practice, 66 (2023)[DOI]
- "OpenAI homepage"[URL]
- Si, Chenglei, "Prompting GPT-3 To Be Reliable", ~ (2022)
- Su, Y., "Collaborating with ChatGPT in argumentative writing classrooms", Assessing Writing (2023)[DOI]
- Thorp H. H., "ChatGPT is fun, but not an author", Science (New York, N.Y.), 379(6630), pp.313~ (2023)[DOI]
- Stokel-Walker, C., "What ChatGPT and generative AI mean for science", Nature, 614(7947), pp.214~216 (2023)[DOI]
- Wei, J., "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models", arXiv, ~ (2022)
- Wood, R., "A systematic review of audience response systems for teaching and learning in higher education: The student experience", Comput. Educ, 153, pp.103896~ (2020)
- Zhai, X., "A review of artificial intelligence(AI)in education from 2010 to 2020", Complexity, 2021, pp.1~18 (2021)
- Zawacki-Richter, O., "Systematic review of research on artificial intelligence applications in higher education – where are the educators?", Int J Educ Technol High Educ, 16, pp.39~ (2019)[DOI]