TF-IDF 임베딩 정보를 이용한 복합 표현의 의항 예측 모델 검토 - 의항 분할에 관여하는 언어 요소의 탐색을 중심으로 -

Abstract

This paper aims to investigate which morphemes are involved in the sense division of multi-word expressions (MWEs) such as 'baega apeuda’, 'naemsaereul punggida', and 'nuneul gamda', which can have both literal and non-compositional meanings, using TF-IDF, a research methodology in the field of artificial intelligence. In Section 2, sentences containing the target MWEs were extracted from a 2-billion-word corpus, and a sample of 14,189 sentences was selected for analysis. The sense of each sentence was manually annotated, and the sentences were morphologically analyzed using a morphological analyzer. In Section 3, the basic concept of TF-IDF is introduced, and the top TF-IDF morphemes for the 7 senses of the target MWEs are examined. In Section 4, the process and results of machine learning on the 14,189 sentences are discussed. It was found that using the top 300 TF-IDF morphemes and the 7 morphemes on either side of the core noun of the MWE as the search span was the most appropriate in terms of machine learning performance and processing speed. It was also found that including grammatical morphemes such as josa and eomi, rather than treating them as stopwords, resulted in a higher F1 score. Additionally, incorporating the grammatical structure of the MWEs (whether they are predicative or modifying) as a feature slightly improved the learning performance. Finally, a linear discriminant analysis (LDA) model was used for the machine learning, and after tuning, an F1 score of 0.8780 was achieved on the evaluation data.