한국어 사전학습모델의 토큰화가 문장 임베딩에 끼치는 영향 분석
The Analysis of the Impact of Tokenization of Korean Pre-trained Model on Sentence Embedding
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
The pre-trained models leading the field of natural language processing these days perform tokenization that do not consider linguistic units, such as Byte-Pair Encoding, WordPiece, or SentencePiece. While these methods alleviate the OOV(Out of Vocabulary) problem, they generate many tokens that have lost their lexical meaning by splitting words into smaller units. This paper analyzes how these tokens affect sentence embedding and ultimately points out the limitations of tokenization of the pre-trained models in this regard. To this end, this study conducts an experiment to determine how tokens interact with sentence embedding depending on whether they preserve their semantics. The interaction between tokens and sentence embedding is measured by Self-Similarity and Intra-Similarity proposed by Ethayarajh(2019). This study found that tokens without semantics show both low Self-Similarity and Intra-Similarity while the other tokens reached a high level in terms of both indicators. Through analysis of the word embedding layer and Self-Attention layer, this study concludes that the former lead to bias in sentence embedding, which is a problem that the pre-trained models inevitably suffer from as long as they continue with existing tokenization.
Abstract
The pre-trained models leading the field of natural language processing these days perform tokenization that do not consider linguistic units, such as Byte-Pair Encoding, WordPiece, or SentencePiece. While these methods alleviate the OOV(Out of Vocabulary) problem, they generate many tokens that have lost their lexical meaning by splitting words into smaller units. This paper analyzes how these tokens affect sentence embedding and ultimately points out the limitations of tokenization of the pre-trained models in this regard. To this end, this study conducts an experiment to determine how tokens interact with sentence embedding depending on whether they preserve their semantics. The interaction between tokens and sentence embedding is measured by Self-Similarity and Intra-Similarity proposed by Ethayarajh(2019). This study found that tokens without semantics show both low Self-Similarity and Intra-Similarity while the other tokens reached a high level in terms of both indicators. Through analysis of the word embedding layer and Self-Attention layer, this study concludes that the former lead to bias in sentence embedding, which is a problem that the pre-trained models inevitably suffer from as long as they continue with existing tokenization.
참고문헌 (18)
- Dzmitry Bahdanau, "Neural Machine Translation by Jointly Learning to Align and Translate", CLR 2015 (2015)
- Danbi Cho, "An Empirical Study of Korean Sentence Representation with Various Tokenization", Electronics, 10, pp.845~ (2021)
- Jacob Devlin, "BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding", Proceedings of NAACL-HLT 2019, pp.4171~4186 (2019)
- Kawin Ethayarajh, "How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings", Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pp.55~65 (2019)
- Tianyu Gao, "SimCSE : Simple Contrastive Learning of Sentence Embeddings", Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp.6894~6910 (2021)
- Ting Jiang, "PromptBERT : Improving BERT Sentence Embeddings with Prompts", Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.8826~8837 (2022)
- Taku Kudo, "SentencePiece : A simple and language independent subword tokenizer and detoknizer for Neural Text Processing", Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing(System Demonstrations), pp.66~71 (2018)
- "KR-BERT: A Small-Scale Korean-Specific Language Model" (2020)[URL]
- Bohan Li, "On the Sentence Embeddings from Pre-trained Language Models", Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp.9119~9130 (2020)
- "RoBERTa : A Robustly Optimized BERT Pretraining Approach" (2019)[URL]
- Nils Reimers., "Sentence-BERT : Sentence Embeddings using Siamese BERT-Network", Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pp.3982~3992 (2019)
- Rico Sennrich, "Neural Machine Translation of Rare Words with Subword Units", Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 1, pp.1715~1725 (2016)
- Jasdeep Singh, "BERT is Not an Interlingua and the Bias of Tokenization", Proceedings of the 2nd Workshop on Deep Learning Approaches for Loew-Resource NLP(DeepLo), pp.47~55 (2019)
- Ashish Vaswani, "Attention is all you need", Proceedings of the 31st International Conference on Neural Information Processing Systems(NIPS'17), pp.6000~6010 (2017)
- Tongzhou Wang, "Understanding contrastive representation learning through alignment and uniformity on the hypersphere", Proceedings of the 37th International Conference on Machine Learning (ICML'20), 119, pp.9929~9939 (2020)
- "Google’s Neural Machine Translation System : Bridging the Gap between Human and Machine Translation" (2016)[URL]
- Yuanment Yan, "ConSERT : A Contrastive Framework for Self-Supervised Sentence Representation Transfer", Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pp.5065~5075 (2021)
- Jiyen Ham, "KorNLI and KorSTS : New Benchmark Datasets for Korean Natural Language Understanding", Findings of the Association for Computational Linguistics : EMNLP, 2020, pp.422~430 (2020)