Embedding Calculus with Nonword Properties Improves Word Sense Disambiguation
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
The present study concerns word sense disambiguation in neural language models using the diagnostic classifiers and hierarchical lexical network. First, we conducted an experiment to see whether the neural models are capable of detecting ambiguous nouns and how they do so. Secondly, we carried out an experiment to verify whether the neural models can identify a specific sense of a lexeme and how they do so. For these experiments, we made use of Word2Vec and FastText as the fixed embedding models and BERT as the contextualized model. In addition, we examined the uniformed and weighted sum method by adding nonword properties (senses). In the case of ambiguity detection, BERT with the general embedding showed better performance than the other models. In regards to sense class detection, BERT with nonword properties showed the best performance on lexemes with numerous senses.
Abstract
The present study concerns word sense disambiguation in neural language models using the diagnostic classifiers and hierarchical lexical network. First, we conducted an experiment to see whether the neural models are capable of detecting ambiguous nouns and how they do so. Secondly, we carried out an experiment to verify whether the neural models can identify a specific sense of a lexeme and how they do so. For these experiments, we made use of Word2Vec and FastText as the fixed embedding models and BERT as the contextualized model. In addition, we examined the uniformed and weighted sum method by adding nonword properties (senses). In the case of ambiguity detection, BERT with the general embedding showed better performance than the other models. In regards to sense class detection, BERT with nonword properties showed the best performance on lexemes with numerous senses.
참고문헌 (29)
- Alt, Christoph, "Probing linguistic features of sentence-level representations in neural relation extraction", ACL 2020 - Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, pp.1534~1545 (2020)
- Arora, Sanjeev, "Linear Algebraic Structure of Word Senses, with Applications to Polysemy", Transactions of the Association for Computational Linguistics, 6, pp.483~495 (2018)
- Bojanowski, Piotr, "Enriching word vectors with subword information", Transactions of the Association for Computational Linguistics, 5, pp.135~146 (2017)
- Buhrmester, Vanessa, "Analysis of explainers of black box deep neural networks for computer vision: A survey", arXiv preprint arXiv:1911.12116, ~ (2019)
- Chi, Ethan A., "Finding Universal Grammatical Relations in Multilingual BERT", arXiv preprint arXiv:2005.04511, ~ (2020)
- Jo, Pyeong-Ok, "A Semantic Hierarchy of Korean Nouns using the Definitions of Words in a Dictionary", Korean Journal of Cognitive Science, 10(4), pp.1~10 (1999)
- Choe, Ho-Seop, "Construction and Application of Large-scale Korean User-Word Intelligent Network", PhD’s, pp.2007
- 최민우, "세종 형태(의미)분석 말뭉치 구축의 실제", 민족문화연구, (48), pp.337~372 (2008), 민족문화연구원
- Conneau, Alexis, "What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties", ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (2018)
- Delmonte, Rodolfo, "Towards Automatic Dialogue Understanding", Proceedings of the International Conference on Spoken Communication (GSCP 2009) (2009)
- Devlin, Jacob, "BERT:Pre-training of deep bidirectional transformers for language understanding", NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, 1, pp.4171~4186 (2019)
- "Eunjeon Project" (2018)[URL]
- Sohn, HoMin, "The Korean Language" (2001), Cambridge University Press
- Hupkes, Dieuwke, "Visualisation and “diagnostic classifiers”, reveal how recurrent and recursive neural networks process hierarchical structure", IJCAI International Joint Conference on Artificial Intelligence (2018)
- Jurafsky, Daniel, "Computational Linguistics, and Speech Recognition" (2019)
- Lee, Sangah, "KR-BERT: A Small-Scale Korean-Specific Language Model" (2020)
- Lesk, Michael, "Automatic sense disambiguation using machine readable dictionaries: How to tell a pine cone from an ice cream cone", Proceedings of the 5th Annual International Conference on Systems Documentation, SIGDOC, pp.24~26 (1986)
- Mikolov, Tomas, "Efficient estimation of word representations in vector space", 1st International Conference on Learning Representations, ICLR 2013 - Workshop Track Proceedings (2013)
- Pires, Telmo, "How multilingual is multilingual BERT?", ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, pp.4996~5001 (2020)
- Altaf, Rahman, "Resolving Complex Cases of Definite Pronouns: The Winograd Schema Challenge" (2012), Association for Computational Linguistics
- Rothe, Sascha, "Autoextend: Combining word embeddings with semantic resources", Computational Linguistics, 43(3), pp.593~617 (2017)
- Schank, Rodger C., "Conceptual dependency: A theory of natural language understanding", Cognitive Psychology (1972)
- Song, Sanghoun, "The Computational Extraction of Semantic Hierarchies for Korean Adjectives", Annual Conference on Human and Language Technology, pp.109~116 (2006)
- 송상헌, "Automatic acquisition of “noun+verb” idiomatic compounds in Korean", 언어연구, 32(1), pp.253~280 (2015), 언어정보연구소[DOI]
- 송상헌, "Selectional Preferences of Korean Verbal Items", 언어연구, 31(2), pp.249~273 (2014), 언어정보연구소[DOI]
- Wang, Alex, "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding", ~ (2018)
- Warstadt, Alex, "“Neural Network Acceptability Judgments", 7, pp.625~641 (2018)
- Wu, Yonghui, "Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation", ~
- Yaghoobzadeh, Yadollah, "Probing for Semantic Classes: Diagnosing the Meaning Content of Word Embeddings", ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, pp.5740~5753 (2019)