한국어 텍스트의 토큰화 방법에 관한 언어학적 연구 - fastText 단어 임베딩을 이용하여 -
A linguistic study on tokenization methods for Korean text
0
조회
0
다운로드
3
인용
0
참고문헌
키워드
초록
This paper aims to provide linguistic understanding of how tokenization methodsaffect the performance of deep-learning-based NLP (Natural Language Processing)models. To this end, we train fastText word embedding models using differenttokenization methods. The methods are categorized by their tokenization unit,eojeol or morpheme, and subword level, syllable (character) or grapheme(subcharacter). Additionally, we apply selective grapheme-decompositionaccording to the type of a morpheme, lexical or grammatical, for the morphemetokenization methods to provide linguistic information to models. Using theseembedding models, we perform a similarity test, an analogy test, and sentimentanalysis. The results show that the selective grapheme-decomposition methods canimprove the performance of a model by preventing duplicated learning of somemorphemes whose morpheme types are different from each other but subwordsare similar or keeping meaningful syllables of lexical morphemes. It is alsosuggested that grapheme-decomposition after morpheme tokenization would beuseful for resolving not only Out-of-Vocabulary problems but also errors of amorphological analyzer.
Abstract
This paper aims to provide linguistic understanding of how tokenization methodsaffect the performance of deep-learning-based NLP (Natural Language Processing)models. To this end, we train fastText word embedding models using differenttokenization methods. The methods are categorized by their tokenization unit,eojeol or morpheme, and subword level, syllable (character) or grapheme(subcharacter). Additionally, we apply selective grapheme-decompositionaccording to the type of a morpheme, lexical or grammatical, for the morphemetokenization methods to provide linguistic information to models. Using theseembedding models, we perform a similarity test, an analogy test, and sentimentanalysis. The results show that the selective grapheme-decomposition methods canimprove the performance of a model by preventing duplicated learning of somemorphemes whose morpheme types are different from each other but subwordsare similar or keeping meaningful syllables of lexical morphemes. It is alsosuggested that grapheme-decomposition after morpheme tokenization would beuseful for resolving not only Out-of-Vocabulary problems but also errors of amorphological analyzer.
참고문헌 (42)
- 김민호, "실질 형태소 기반 부분단어 정보를 이용한 단어 임베딩", 한국정보과학회 학술발표논문집, pp.569~571 (2018), 한국정보과학회
- 박진호, "딥러닝 기반 자연어 처리에서 도메인 지식의 역할", 인공지능인문학연구, 4, pp.134~166 (2019), 인문콘텐츠연구소[DOI]
- 배성봉, "한자어 인지와 학습에서 의미투명성의 효과", 교육심리연구, 26(2), pp.607~620 (2012), 한국교육심리학회[DOI]
- 송현제, "서브워드 토큰화와 한국어 형태소 분석기", 정보과학회지, 39(4), pp.15~20 (2021), 한국정보과학회
- 윤준영, "부분단어와 품사 태깅 정보를 활용한 형태소 기반의 한국어 단어 벡터 생성", 정보과학회논문지, 47(4), pp.395~403 (2020), 한국정보과학회[DOI]
- 이광오, "표기 체계와 시각적 단어 인지:한자어의 인지에서 형태소의 표상과 처리", 한국심리학회지: 인지 및 생물, 19(4), pp.313~327 (2007), 한국인지및생물심리학회[DOI]
- 이기창, "한국어 임베딩: 자연어 처리 모델의 성능을 높이는 핵심 비결 Word2Vec에서 ELMo, BERT까지" (2019), 에이콘
- 이동준, "형태소 기반 효율적인 한국어 단어 임베딩", 정보과학회논문지, 45(5), pp.444~450 (2018), 한국정보과학회[DOI]
- 정유남, "딥러닝을 이용한 노동신문의 이념적 어휘 연구", 한국어학, 88, pp.213~245 (2020), 한국어학회[DOI]
- 조단비, "부분 단어 토큰화 기법을 이용한 뉴스 기사 정치적 편향성 자동 분류 및 어휘 분석", 정보처리학회 논문지, 10(1), pp.1~8 (2021), 한국정보처리학회[DOI]
- 조현수, "FastText를 적용한 한국어 단어 임베딩", 한국정보과학회 학술발표논문집, pp.705~707 (2017), 한국정보과학회
- Ansdell, M., "Scientific domain knowledge improves exoplanet transit classification with deep learning", The Astrophysical journal letters, 869(1), pp.L7~ (2018)
- Bojanowski, P., "Enriching word vectors with subword information", Transactions of the Association for Computational Linguistics, 5, pp.135~146 (2017)
- Carreiras, M., "Syllable frequency and visual word recognition in Spanish", Journal of memory and language, 32(6), pp.766~780 (1993)
- Cho, D., "An Empirical Study of Korean Sentence Representation with Various Tokenizations", Electronics, 10(7), pp.845~ (2021)
- Choi, S., "A syllable-based technique for word embeddings of Korean words", arXiv preprint arXiv:1708.01766, ~ (2017)
- Devlin, J., "Bert: Pre-training of deep bidirectional transformers for language understanding", arXiv preprint arXiv:1810.04805, ~ (2018)
- Finkelstein, L., "Placing search in context: The concept revisited", Proceedings of the 10th international conference on World Wide Web, pp.406~414 (2001)
- Gage, P., "A new algorithm for data compression", C Users Journal, 12(2), pp.23~38 (1994)
- Grefenstette, G., "Syntactic Wordclass Tagging", pp.117~133 (1999), Springer
- Hochreiter, S., "Long short-term memory", Neural computation, 9(8), pp.1735~1780 (1997)
- Kim, B., "What changes can large-scale language models bring? intensive study on hyperclova: Billions-scale korean generative pretrained transformers", arXiv preprint arXiv:2109.04650, ~ (2021)
- Kim, G., "Enhancing Korean Named Entity Recognition with Linguistic Tokenization Strategies", IEEE Access (2021)
- Kingma, D. P., "Adam: A method for stochastic optimization", arXiv preprint arXiv:1412.6980, ~ (2014)
- Lee, S., "KR-BERT: A Small-Scale Korean-Specific Language Model", arXiv preprint arXiv:2008.03979, ~ (2020)
- Levy, O., "Linguistic regularities in sparse and explicit word representations", Proceedings of the eighteenth conference on computational natural language learning, pp.171~180 (2014)
- Kudo, T., "Subword regularization: Improving neural network translation models with multiple subword candidates", arXiv preprint arXiv:1804.10959, ~ (2018)
- Kudo, T., "Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing", arXiv preprint arXiv:1808.06226, ~ (2018)
- Kwon, O., "Handling Out-Of-Vocabulary Problem in Hangeul Word Embeddings", Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp.3213~3221 (2021)
- Kwon, Y., "The role of orthographic syllable frequency in the syllable frequency effect : Evidence from Korean", Perceptual and motor skills, 120(1), pp.95~109 (2015)
- Mathey, S., "The nature of the syllabic neighbourhood effect in French", Acta Psychologica, 123(3), pp.372~393 (2006)
- Mikolov, T., "Distributed representations of words and phrases and their compositionality", Advances in neural information processing systems, pp.3111~3119 (2013)
- Muralidhar, N., "Incorporating prior domain knowledge into deep neural networks", 2018 IEEE international conference on big data (big data), pp.36~45 (2018)
- Park, S., "Subword-level word vector representations for Korean", Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, 1, pp.2429~2438 (2018)
- Park, K., "An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks", arXiv preprint arXiv:2010.02534, ~ (2020)
- Park, C., "Should we find another model?: Improving Neural Machine Translation Performance with ONE-Piece Tokenization Method without Model Modification", Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers, pp.97~104 (2021)
- Park, S., "Klue: Korean language understanding evaluation", arXiv preprint arXiv:2105.09680, ~ (2021)
- Sennrich, R., "Neural machine translation of rare words with subword units", arXiv preprint arXiv:1508.07909, ~ (2015)
- Stratos, K., "A sub-character architecture for Korean language processing", arXiv preprint arXiv:1707.06341, ~ (2017)
- Vaswani, A., "Attention is all you need", Advances in neural information processing systems, pp.5998~6008 (2017)
- Vo, K., "Combination of domain knowledge and deep learning for sentiment analysis of short and informal messages on social media", International Journal of Computational Vision and Robotics, 9(5), pp.458~448 (2019)
- Wu, Y., "Google's neural machine translation system: Bridging the gap between human and machine translation", arXiv preprint arXiv:1609.08144, ~ (2016)