본문 바로가기
KERD
논문KCI 등재

Evaluating the Performance of ChatGPT on the Secondary School English Teacher Hiring Examination

이주원|최선희|이제영|황영|최소담
한국영어교과교육학회 (The Korea English Education Society)|2023.11|영어교과교육 (Journal of the Korea English Education Society)|제22권 제4호|pp. 73-95|DOI 10.18649/jkees.2023.22.4.73

0

조회

0

다운로드

0

인용

0

참고문헌

키워드

ChatGPTlarge language modelperformance analysisteacher hiring examination챗GPT대용량 언어 모델수행 분석교원 임용 시험

초록

The purpose of the present study is to conduct a comprehensive evaluation and analysis of the performance of OpenAI’s ChatGPT, a language model based on GPT-4 architecture, on the Secondary School English Teacher Hiring Examination. For this purpose, the current study employed ChatGPT-4, the most recent model as its primary research instrument. A total of 23 questions from the 2023 hiring exam were fed into ChatGPT for response generation. The responses elicited from ChatGPT were then analyzed by a panel of five experts according to their respective fields of expertise. The results of the analyses highlighted both the potential and limitations of ChatGPT as a pedagogical tool. The results also identified the AI’s propensity to simplify complex or abstract human knowledge into narrower or more technical concepts. These observations underscore the limitations of current AI in fully emulating human comprehension and its capacity to interpret nuances in knowledge. Nonetheless, these findings also suggest a direction for future AI development and reinforce the idea of AI as a supplementary learning tool rather than a complete replacement of human teachers.

Abstract

The purpose of the present study is to conduct a comprehensive evaluation and analysis of the performance of OpenAI’s ChatGPT, a language model based on GPT-4 architecture, on the Secondary School English Teacher Hiring Examination. For this purpose, the current study employed ChatGPT-4, the most recent model as its primary research instrument. A total of 23 questions from the 2023 hiring exam were fed into ChatGPT for response generation. The responses elicited from ChatGPT were then analyzed by a panel of five experts according to their respective fields of expertise. The results of the analyses highlighted both the potential and limitations of ChatGPT as a pedagogical tool. The results also identified the AI’s propensity to simplify complex or abstract human knowledge into narrower or more technical concepts. These observations underscore the limitations of current AI in fully emulating human comprehension and its capacity to interpret nuances in knowledge. Nonetheless, these findings also suggest a direction for future AI development and reinforce the idea of AI as a supplementary learning tool rather than a complete replacement of human teachers.

참고문헌 (32)

  1. Adiguzel, T., "Revolutionizing education with AI : Exploring the transformative potential of ChatGPT", Contemporary Educational Technology, 15(3), pp.ep429~ (2023)
  2. 안유영, "Performance of ChatGPT 3.5 on CSAT: Its Potential as a Language Learning and Assessment Tool", 영어교과교육, 22(2), pp.119~145 (2023), 한국영어교과교육학회
  3. Barrot, J. S., "Using ChatGPT for second language writing : Pitfalls and potentials", Assessing Writing, 57, pp.100745~ (2023)[DOI]
  4. Biswas, S., "ChatGPT and the future of medical writing", Radiology, 307(2), pp.e223312~ (2023)[DOI]
  5. Som Biswas, "Role of ChatGPT in Gaming: According to ChatGPT", SSRN Electronic Journal (2023), Elsevier BV[DOI]
  6. 차수미, "영어교육 관련 AI 챗봇 연구 논문의 동향 분석", 영어교과교육, 20(1), pp.203~225 (2021), 한국영어교과교육학회[DOI]
  7. 최윤희, "영어 작문 과정에 활용한 ChatGPT에 대한 한국인 예비영어교사의 인식 탐색", 영어교과교육, 22(2), pp.243~262 (2023), 한국영어교과교육학회
  8. Ellis, R., "Second language acquisition" (1997), Oxford University Press
  9. Fuchs, K., "Exploring the opportunities and challenges of NLP models in higher education : Is ChatGPT a blessing or a curse", Frontiers in Education, 8, pp.1166682~ (2023)[DOI]
  10. Panagiotis Giannos, "Performance of ChatGPT on UK Standardized Admission Tests: Insights From the BMAT, TMUA, LNAT, and TSA Examinations", JMIR Medical Education, 9, pp.e47737~ (2023), JMIR Publications Inc.[DOI]
  11. Gilson, A., ". How does ChatGPT perform on the United States medical licensing examinations? The implications of large language models for medical education and knowledge assessment", JMIR Medical Education, 9, pp.e45312~ (2023)[DOI]
  12. 현주은, "영어학습도구로서 인공지능 스피커의 분석 및 시사점", 영어영문학, 24(1), pp.201~219 (2019), 미래영어영문학회[DOI]
  13. Jaeho Jeon, "Large language models in education: A focus on the complementary relationship between human teachers and ChatGPT", Education and Information Technologies, 28(12), pp.15873~15892 (2023), Springer Science and Business Media LLC[DOI]
  14. Kasneci, E., "ChatGPT for good? On opportunities and challenges of large language models for education", Learning and Individual Differences, 103, pp.102274~ (2023)[DOI]
  15. 김혜영, "영어교과 보조 도구로서의 AI 챗봇 분석 연구", 학습자중심교과교육연구, 19(1), pp.89~110 (2019), 학습자중심교과교육학회[DOI]
  16. 김성우, "인공지능기술 활용 대화형 영어 학습 앱을 적용한 초등영어 말하기 수업 분석: 학습자 몰입 및 수업에 대한 학생과 교사의 인식을 중심으로", 초등영어교육, 26(3), pp.177~202 (2020), 한국초등영어교육학회[DOI]
  17. Kung, T. H., "Performance of ChatGPT on USMLE : Potential for AI-assisted medical education using large language models", PLOS Digital Health, 2(2), pp.e0000198~ (2023)[DOI]
  18. "ChatGPT’s scientific Writing : A case study on traffic safety" (2023)[URL]
  19. 권서경, "생성형 인공지능 ChatGPT의 영어 독해 문항 풀이 능력 탐색", 영어교과교육, 22(2), pp.147~172 (2023), 한국영어교과교육학회
  20. 이제영, "A Meta-analysis of the Effects of Using AI Chatbot in Korean EFL Education", 영어영문학연구, 48(1), pp.213~243 (2022), 대한영어영문학회[DOI]
  21. Li, B., "Exploring the multilingual applications of ChatGPT : Uncovering language learning affordances in YouTuber videos", International Journal of Computer-Assisted Language Learning and Teaching, 13(1), pp.1~22 (2023)[DOI]
  22. OpenAI, "GPT-4 technical report. arXiv, 2023-08774", ~ (2023)
  23. Paul, J., "ChatGPT and consumers : Benefits, pitfalls and future research agenda", International Journal of Consumer Studies, 47(4), pp.1213~1225 (2023)[DOI]
  24. Perez-Marin, D., "Conversational agents and natural language interaction: Techniques and effective practices", Information Science Reference, ~ (2011)
  25. 신동광, "영어 쓰기 능력 향상을 위한 AI 챗봇 활용 방안 탐색", 교원교육, 35(1), pp.41~55 (2019), 교육연구원[DOI]
  26. Sutrisno, S., "Exploring the potential of ChatGPT to improve customer service in MSMEs", Innovative : Journal Of Social Science Research, 3(2), pp.653~664 (2023)[DOI]
  27. Thorp, H. H., "ChatGPT is fun, but not an author", Science, 279(6630), pp.313~ (2023)[DOI]
  28. Weizenbaum, J., "ELIZA : A computer program for the study of natural language communication between man and machine", Communications of the ACM, 9(1), pp.36~45 (1966)[DOI]
  29. Wu, H., "ChatGPT or Grammarly? Evaluating ChatGPT on grammatical error correction benchmark", arXiv, 2023, 13648v1, ~ (2023)
  30. Da Yan, "Impact of ChatGPT on learners in a L2 writing practicum: An exploratory investigation", Education and Information Technologies, 28(11), pp.13943~13967 (2023), Springer Science and Business Media LLC[DOI]
  31. Hong Yang, "How I use ChatGPT responsibly in my teaching", Nature (2023), Springer Science and Business Media LLC[DOI]
  32. Zografos, G., "Augmented intelligence and intelligent tutoring systems", pp.270~280 (2023), Springer

이 간행물의 다른 논문 (12건)

Examining Satisfaction Levels with Teaching Methods in University General English Courses during the COVID-19 and Post-COVID-19 Era이은혜, 안혜성, 박매란pp. 1-26The Roles of Individual Differences and Linguistic Features in EFL Argumentative Writing김민경pp. 27-47Building a Wordlist from 2006-2021 Mock and Official College Scholastic Ability Tests (CSAT) Uncovered in the 2022 Revised National Curriculum of English전문기, 이혜진, 류지수pp. 49-72The Cognitive Effects of MALL Combined with Feedback in EFL/ESL: A Meta-analysis윤경호, 이동주pp. 97-123초등영어 학습부진아의 학습 고충에 대한 이해: 근거이론적 접근곽화원, 이상기pp. 125-148
원문 보기