사람과 AI 모델 간의 화행 주석 개선 방향 연구

Abstract

In this study, we aim at improving the quality of speech act annotation by identifying and analyzing the annotation agreement between humans and the GPT-4 model to identify the categories of speech acts that the GPT-4 model frequently misclassifies. For this purpose, we selected a total of 33,138 sentences from the National Institute of Korean Language’s Messenger corpus (2022). The annotation was performed by human annotators and the GPT-4 model, and the classification accuracy was evaluated using the RoBERTa model. The accuracy score for the human-annotated messenger corpus is 85.50, while the accuracy score for the cases where both humans and the GPT-4 model annotated identically is 96.32. In particular, the category of speech acts that the GPT-4 model most frequently misclassifies is “sarcasm/humor.” This is likely due to highly frequent expressions such as “ㅋㅋㅋ ” (lol) in the messenger corpus, which causes the model to incorrectly analyze even “Statement” as “sarcasm/humor.” For example, even when a human annotator classified the utterance “I have snacks and ramen. ㅋㅋㅋ” as “Statement”, the GPT-4 model misclassified it as “sarcasm/humor.” In an experiment conducted using only the sentences where the model and human annotators agreed, the accuracy score reached 95.32. Instead of merely discussing annotator agreement for newly proposed corpora, it is necessary to distinguish speech acts that are difficult to annotate from those that are relatively easy. Furthermore, within these categories, it is essential to select sentences that are challenging to automatically annotate and conduct research focused on more challenging areas.