생성 인공지능을 활용한 영문학 교육의 가능성 연구: Gemini Pro, GPT-3.5, GPT-4의 패러디 과제 평가 분석을 중심으로
Exploring the Pedagogical Potential of Generative AI in English Literature Education: Comparative Assessment of Parody Assignments Through Gemini Pro, GPT-3.5, and GPT-4
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
This study investigates how effectively generative AI can provide feedback on creative writing for English literature education. In order to examine the reliability and validity of the automated feedback, 40 movie parody assignments written by 40 English major undergraduates were evaluated by three AI models (Gemini 1.0 Pro, GPT-3.5-turbo, and GPT-4-turbo) with four constructs. Statistical analysis of the three models’ scores, generated at four temperature settings (0, 0.4, 0.7, 1.0) with 30 iterations per setting, are as follows: First, all three models showed high reliability in scoring on a 0 to 5 scale. Second, rank-based scoring analysis revealed differences in reliability among the models. Gemini 1.0 Pro demonstrated high reliability only at 0 temperature, whereas GPT-4 maintained consistent reliablity across all temperature settings. Third, scoring validity varied significantly among models. Validity assessments through the review of student assignments and the analysis of rank-based scoring contrasted the limited validity of Gemini 1.0 Pro with the high validity of GPT-4’s consistent reliability scoring. However, GPT-4’s tendency toward holistic scoring should be considered. These findings suggest that all three models, which can consistently provide general-level feedback, would be limited in providing optimized feedback. The significance of this study lies in its empirical validation of generative AI’s potential as a feedback provider in creative writing education.
Abstract
This study investigates how effectively generative AI can provide feedback on creative writing for English literature education. In order to examine the reliability and validity of the automated feedback, 40 movie parody assignments written by 40 English major undergraduates were evaluated by three AI models (Gemini 1.0 Pro, GPT-3.5-turbo, and GPT-4-turbo) with four constructs. Statistical analysis of the three models’ scores, generated at four temperature settings (0, 0.4, 0.7, 1.0) with 30 iterations per setting, are as follows: First, all three models showed high reliability in scoring on a 0 to 5 scale. Second, rank-based scoring analysis revealed differences in reliability among the models. Gemini 1.0 Pro demonstrated high reliability only at 0 temperature, whereas GPT-4 maintained consistent reliablity across all temperature settings. Third, scoring validity varied significantly among models. Validity assessments through the review of student assignments and the analysis of rank-based scoring contrasted the limited validity of Gemini 1.0 Pro with the high validity of GPT-4’s consistent reliability scoring. However, GPT-4’s tendency toward holistic scoring should be considered. These findings suggest that all three models, which can consistently provide general-level feedback, would be limited in providing optimized feedback. The significance of this study lies in its empirical validation of generative AI’s potential as a feedback provider in creative writing education.
참고문헌 (31)
- 구영은, "Chatgpt를 활용한 독일어 초급 학습자의 작문 연습에관한 연구", 독일언어문학, 100, pp.47~69 (2023)
- 김기호, "대학 글쓰기 교양교육에서 ‘창의성’ 관련 연구의 성과와 과제", 어문학, 130, pp.261~293 (2015)
- 김미영, "패러디를 활용한 소설교육의 방법", 한국문학이론과 비평, 9(1), pp.315~337 (2005)
- 김성문, "창의적 글쓰기를 위한 시조의 활용 방안 고찰", 어문론집, 59, pp.123~146 (2014)
- 손혜숙, "대학 글쓰기에서의 패러디 글쓰기 교육방법 연구", 리터러시 연구, 11, pp.77~98 (2015)
- 안현수, "Ai 기반 피드백의 교육적 활용 가능성 탐색", 이중언어학, 96, pp.63~95 (2024)
- 이지연, "챗gpt를 활용한 영어 작문 평가", 현대영미어문학, 42(1), pp.189~212 (2024)
- 장성민, "생성 인공지능 기술은 인간의 쓰기 능력을 대신할 수 있는가?-포스트 챗gpt 시대의 작문 평가 담론을 위한 예비적 논의", 국어교육, 181, pp.119~160 (2023)
- 조지현, "크리티컬 리터러시 기반의 영문학 강의 콘텐츠 개발:『오만과 편견』과 패러디 텍스트를 중심으로", 영미문학교육, 20(3), pp.91~116 (2016)
- 조희정, "패러디를 통한 영시 교육 방법의 모색: 『그의 수줍은 연인에게』의 개작시들을 중심으로", 영어영문학연구, 58(3), pp.229~249 (2016)
- 최강민, "패러디 글쓰기의 서사 전략과 발상의 전환", 교양학연구, 12, pp.91~124 (2020)
- 최석무, "크리티컬 리터러시를 활용한 “백설공주” 읽기교육—원작과영화, 패러디 작품을 중심으로", 영어영문학, 55(5), pp.885~906 (2009)
- 최선희, "이공계생의 글쓰기 흥미와 창의성을 위한 글쓰기 수업 방안-패러디를 활용하여", 어문학, 148, pp.341~371 (2020)
- 최홍원, "고전시가의 패러디를 통한 창의성 교육의 실제와 과제", 한어문교육, 24, pp.5~36 (2011)
- 한승우, "대학 글쓰기 수업에서의 패러디 활용", 리터러시 연구, 15, pp.11~34 (2016)
- Attali, Yigal, "Handbook of Automated Essay Evaluation", pp.181~198 (2013), Routledge
- Bachman, L. F., "Language Testing in Practice:Designing and Developing Useful Language Tests" (1996), Oxford UP
- Burstein, Jill C, "Automated Essay Scoring: A Cross-Disciplinary Perspective", pp.107~115 (2003), Lawrence Erlbaum Associates
- Chen, Hong, "Storyer: Automatic Story Evaluation Via Ranking, Rating and Reasoning", Journal of Natural Language Processing, 30(1), pp.243~249 (2023)
- Dai, Wei, "Can Large Language Models Provide Feedback to Students? A Case Study on Chatgpt", 2023 IEEE International Conference on Advanced Learning Technologies (ICALT), pp.323~325 (2023)
- Dikli, Semire, "An Overview of Automated Scoring of Essays", The Journal of Technology, Learning and Assessment, 5(1), pp.4~35 (2006)
- Foltz, Peter W., "Handbook of Automated Essay Evaluation", pp.68~88 (2013), Routledge
- Hattie, John, "The Power of Feedback", Review of Educational Research, 77(1), pp.81~112 (2007)
- Hutcheon, Linda, "A Theory of Parody: The Teachings of Twentieth-Century Art Forms" (2000), U of Illinois P
- Koo, Terry K, "A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research", Journal of Chiropractic Medicine, 15(2), pp.155~163 (2016)
- GPT-4 Technical Report, "OpenAI", ~ (2023)
- Piper, Andrew, "Narrative Theory for Computational Narrative Understanding", Proceedings of the 2021Conference on Empirical Methods in Natural Language Processing, pp.298~311 (2021)
- Schultz, Matthew T., "Handbook of Automated Essay Evaluation", pp.89~98 (2013), Routledge
- Shi, Huawei, "A Systematic Review of Ai-Based Automated Written Feedback Research", ReCALL, 36(2), pp.187~209 (2024)
- Yancey, Kevin P., "Rating Short L2 Essays on the CEFR Scale with Gpt-4", Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pp.576~584 (2023)
- Zhu, Lixing, "Are NLP Models Good at Tracing Thoughts: An Overview of Narrative Understanding", Findings of the Association for Computational Linguistics: EMNLP 2023, pp.10098~10121 (2023)