텍스트마이닝을 위한 한국어 범용 불용어 목록 연구

Abstract

Due to the strong applicability of the Stopwords processing, the selection range of the words can vary depending on the mediums of delivery of the speech and the subject of the text. Thus, This paper used a large-scale spoken corpus to extract common Korean Stop words, considering that language is divided into spoken and written languages. Also, the linguistic analysis method was used as an auxiliary method, considering that there is a certain limit to relying only on statistical methods in extracting stop words. This paper extracted 576 Stopwords firstly using five statistical methods including Text Frequency method (TF), Document Frequency method (DF), Text Frequency-Inverse Document Frequency method (TF-IDF), Information Entropy method (EC), Term Frequency-Inverse Word Frequency (TF-IWF) and etc. Then 149 Korean spoken words were extracted secondly using linguistic analysis method. Lastly, 149 Korean spoken words and 324 Korean written words suggested in previous studies were compared and analyzed to present a list of general-purpose Korean Stopwords suitable for spoken and written text mining.