생성형 AI의 문법적 능력에 대한 국어학적 연구 - 형태 통사적 특성을 중심으로 -
A study on the grammatical competence of generative AI - Focusing on Korean morpho-syntactic features -
0
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
This study aims to determine whether generative AI has grammatical skills at the level of native Korean speakers. For this purpose, a grammatical understanding ability test and a production ability test were conducted. In Chapter 2, we reviewed previous research related to language ability evaluation of generative AI and presented an evaluation method. In Chapter 3, the grammatical ability of generative AI for ‘endings’ and ‘negative sentences’ was tested in terms of understanding and production. This work could improve the performance of domestic and foreign language models. This is because it summarizes areas where Korean language study can contribute, such as fine-tuning specific errors in terms of Korean grammatical ability. And conversely, it can be said that the possibility of using generative AI services in Korean language education and research was also discussed. The results of this study can be summarized into the following three points. First, these models generally showed a much higher percentage of correct answers in the grammaticality judgment test than in the correction. This is natural, assuming that generative AI is a Korean language learner. This provides an important clue to show that the current level of Korean language proficiency of generative AI has a long way to go and to gauge the direction for raising the level of Korean proficiency of generative AI to that of a native language speaker. Second, at least for the questions evaluated in this study, CLOVA X, developed by Naver, a domestic company, showed the highest accuracy rate. However, even CLOVA X had a very low accuracy rate in evaluating grammatical production ability for endings. This will be an important clue in assessing and improving the level of Korean language proficiency of current generative AI. Third, although ChatGPT 4.0 is the only paid version among generative AIs, it is overall superior to ChatGPT 3.5 in evaluating Korean grammar skills, but the difference is not noticeable.
Abstract
This study aims to determine whether generative AI has grammatical skills at the level of native Korean speakers. For this purpose, a grammatical understanding ability test and a production ability test were conducted. In Chapter 2, we reviewed previous research related to language ability evaluation of generative AI and presented an evaluation method. In Chapter 3, the grammatical ability of generative AI for ‘endings’ and ‘negative sentences’ was tested in terms of understanding and production. This work could improve the performance of domestic and foreign language models. This is because it summarizes areas where Korean language study can contribute, such as fine-tuning specific errors in terms of Korean grammatical ability. And conversely, it can be said that the possibility of using generative AI services in Korean language education and research was also discussed. The results of this study can be summarized into the following three points. First, these models generally showed a much higher percentage of correct answers in the grammaticality judgment test than in the correction. This is natural, assuming that generative AI is a Korean language learner. This provides an important clue to show that the current level of Korean language proficiency of generative AI has a long way to go and to gauge the direction for raising the level of Korean proficiency of generative AI to that of a native language speaker. Second, at least for the questions evaluated in this study, CLOVA X, developed by Naver, a domestic company, showed the highest accuracy rate. However, even CLOVA X had a very low accuracy rate in evaluating grammatical production ability for endings. This will be an important clue in assessing and improving the level of Korean language proficiency of current generative AI. Third, although ChatGPT 4.0 is the only paid version among generative AIs, it is overall superior to ChatGPT 3.5 in evaluating Korean grammar skills, but the difference is not noticeable.
참고문헌 (29)
- "인공지능 AI 말평", 국립국어원[URL]
- 김재희, "인공지능 대화형 에이전트의 공손성 연구-ChatGPT를 중심으로", 언어사실과 관점, 60, pp.9~36 (2023)
- 권서경, "생성형 인공지능 ChatGPT의 영어 독해 문항 풀이 능력 탐색", 영어교과교육, 22(2), pp.147~172 (2023)
- 남기심, "새로 쓴 표준 국어문법론" (2021), 한국문화사
- 백미경, "ChatGPT의 한국어 능력 평가에 대한 연구-한국어능력시험 읽기 영역 및 관용구를 중심으로", 언어사실과 관점, 59, pp.279~308 (2023)
- 송상헌, "딥러닝 언어모형의 평가와 언어학", 언어와 정보 사회, 45, pp.169~191 (2022)
- 송상헌, "인공지능 언어모델의 상식적 판단", 국어학, 105, pp.375~400 (2023)
- 신동광, "내용중심 영어 교수 학습의 도구로서 ChatGPT의 활용 가능성 탐색", 영어교과교육, 22(1), pp.171~192 (2023)
- 문장 문법성 판단을 위한 기초 자료 구축, "안희돈", 국립국어원, ~ (2019)
- 이명원, "ChatGPT의 영어 문법성 판단에 근거한 타당성 검증", 영어평가, 18(2), pp.213~241 (2023)
- 이지은, "인공지능 언어 모델의 절 연결 능력에 대한 일고찰 : 터보 및 다빈치 모델을 대상으로", 언어과학연구, 105, pp.27~61 (2023)
- 정한데로, "‘생성 AI 화자’의 단어 형성-<ChatGPT>, <Bard>, <CLOVA X>를 대상으로-", 한말연구, 64(54), pp.1~26 (2023)
- Ahn, Y., "Performance of ChatGPT 3.5 on CSAT: Its Potential as a Language Learning and Assessment Tool", Journal of the Korea English Education Society, 22(2), pp.119~145 (2023)
- Bachman L., "Fundamental Considerations in Language Testing" (1990), Oxford University Press
- Casal, J. E., "Can linguists distinguish between ChatGPT", AI and human writing? : A study of research ethics and academic publishing, 2023, pp.2(3), 100068~
- Chomsky, N., "Syntactic structures.(Janua linguarum, 4.)" (1957), Mouton
- Chomsky, N., "Aspects of the theory of syntax" (1965), M. I. T. Press. C
- "Noam Chomsky: The False Promise of ChatGPT" (2023)[URL]
- "A test of artificial intelligence, senstein" (2023)[URL]
- Fang, T., "Is chatgpt a highly fluent grammatical error correction system. A comprehensive evaluation", CoRR, abs, 2023
- Kohnke, L., "ChatGPT for language teaching and learning", RELC Journal, 54(2), pp.537~550 (2023)
- Krashen, S. D., "The Input Hypothesis: Issues and Implications" (1985), Longman
- Lau, J. H., "Grammaticality, acceptability, and probability : A probabilistic view of linguistic knowledge", Cognitive science, 41(5), pp.1202~1241 (2017)
- Wang, A., "Superglue : A stickier benchmark for general-purpose language understanding systems", Advances in neural information processing systems, 32 (2019)
- Warstadt, A., "Neural network acceptability judgments", Transactions of the Association for Computational Linguistics, 7, pp.625~641 (2019)
- "Does ChatGPT resemble humans in language use?" (2023)[URL]
- "ChatGPT grammaticality judgement" (2023)[URL]