Hierarchical Sentiment Classification for Agglutinative Languages: Enhancing Annotation Consistency with Human-in-the-Loop and Collective Intelligence

Abstract

High-quality sentiment analysis datasets are critical for enhancing the performance of natural language processing (NLP) systems, especially for morphologically complex agglutinative languages. This study aims to improve the consistency and accuracy of sentiment annotation by considering the unique linguistic characteristics of such languages. To this end, we propose a hybrid annotation framework that combines Hierarchical Sentiment Voting (HSV), which organizes sentiment classification into multiple levels, with a Human-in-the-Loop (HITL) mechanism that selectively applies human validation to low-confidence annotations. The study focuses on agglutinative languages, including Korean, and constructs a dataset of over 1.5 million text samples collected from various online sources such as product reviews, movie critiques, and community comments. Experimental results show that the proposed method significantly improves inter-annotator agreement and model performance, particularly in contexts requiring fine-grained sentiment distinctions. This research empirically demonstrates the importance of structured, human-refined annotation for building reliable sentiment datasets in agglutinative languages and presents a practical framework that can support the development of accurate sentiment analysis models.