Cats Climb Entails Mammals Move: Preserving Hyponymy in Compositional Distributional Semantics
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
To give vector-based representations of meaning more structure, an approach proposed in Piedeleu et al. (2015); Sadrzadeh et al. (2018); Bankova et al. (2018) is to use positive semidefinite (psd) matrices. These allow us to model similarity of words as well as the hyponymy or is-a relationship. To compose words to form phrases and sentences, we may represent adjectives, verbs, and other functional words as multilinear, positivity preserving maps, following the compositional distributional approach introduced in Coecke et al. (2010) and extended to the realm of psd matrices in Piedeleu et al. (2015), but it is not clear how to learn representations of functional words when working with psd matrices. In this paper, we introduce a generic way of composing the psd matrices corresponding to words. We propose that psd matrices for verbs, adjectives, and other functional words be lifted to completely positive (CP) maps that match their grammatical type. This lifting is carried out by our composition rule called Compression, Compr. In contrast to previous composition rules like Fuzz and Phaser (Coecke and Meichanetzidis, 2020) (a.k.a. KMult and BMult (Lewis, 2019a)), Compr preserves hyponymy. Mathematically, Compr is itself a CP map, and is therefore linear and generally non-commutative. We give a number of proposals for the structure of Compr, based on spiders, cups, and caps, and generate a range of composition rules. We test these rules on sentence entailment datasets from Kartsaklis and Sadrzadeh (2016), and see some improvements over the performance of Fuzz and Phaser. We go on to estimate the parameters of a simplified form of Compr based on entailment information from the aforementioned datasets, and find that whilst this learnt operator does not consistently outperform previously proposed mechanisms, it is competitive and has the potential to improve with the use of a less simplified version.
Abstract
To give vector-based representations of meaning more structure, an approach proposed in Piedeleu et al. (2015); Sadrzadeh et al. (2018); Bankova et al. (2018) is to use positive semidefinite (psd) matrices. These allow us to model similarity of words as well as the hyponymy or is-a relationship. To compose words to form phrases and sentences, we may represent adjectives, verbs, and other functional words as multilinear, positivity preserving maps, following the compositional distributional approach introduced in Coecke et al. (2010) and extended to the realm of psd matrices in Piedeleu et al. (2015), but it is not clear how to learn representations of functional words when working with psd matrices. In this paper, we introduce a generic way of composing the psd matrices corresponding to words. We propose that psd matrices for verbs, adjectives, and other functional words be lifted to completely positive (CP) maps that match their grammatical type. This lifting is carried out by our composition rule called Compression, Compr. In contrast to previous composition rules like Fuzz and Phaser (Coecke and Meichanetzidis, 2020) (a.k.a. KMult and BMult (Lewis, 2019a)), Compr preserves hyponymy. Mathematically, Compr is itself a CP map, and is therefore linear and generally non-commutative. We give a number of proposals for the structure of Compr, based on spiders, cups, and caps, and generate a range of composition rules. We test these rules on sentence entailment datasets from Kartsaklis and Sadrzadeh (2016), and see some improvements over the performance of Fuzz and Phaser. We go on to estimate the parameters of a simplified form of Compr based on entailment information from the aforementioned datasets, and find that whilst this learnt operator does not consistently outperform previously proposed mechanisms, it is competitive and has the potential to improve with the use of a less simplified version.
참고문헌 (67)
- Bankova, Dea, "Graded hyponymy for compositional distributional semantics", Journal of Language Modelling, 6(2), pp.225~260 (2018)
- Baroni, Marco, "Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space", Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pp.1183~1193 (2010), Association for Computational Linguistics
- Barwise, J, "Philosophy, language, and artificial intelligence", pp.241~301 (1981), Springer
- Bowman, Samuel, "Recursive neural networks can learn logical semantics", Proceedings of the 3rd workshop on continuous vector space models and their compositionality, pp.12~21 (2015)
- Bowman, Samuel R., "A large annotated corpus for learning natural language inference", Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp.632~642 (2015), Association for Computational Linguistics
- Clarke, Daoud, "Context-theoretic semantics for natural language: an overview", Proceedings of the Workshop on Geometrical Models of Natural Language Semantics, pp.112~119 (2009), Association for Computational Linguistics
- Coecke, B, "The mathematics of text structure", arXiv:1904.03478, ~ (2019)
- Coecke, B., "Lambek vs. Lambek: Functorial vector space semantics and string diagrams for Lambek calculus", Ann Pure Appl Log, 164(11), pp.1079~1100 (2013)
- Coecke, B., "Mathematical foundations for a compositional distributional model of meaning", Lambek Festschrift, special issue of Linguistic Analysis (2010)
- Coecke, Bob, "Meaning updating of density matrices", Journal of Applied Logics, 2631(5), pp.745~ (2020)
- Conneau, Alexis, "Supervised learning of universal sentence representations from natural language inference data", Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp.670~680 (2017)
- Cooper, R., "Fracas: A framework for computational semantics. Deliverable D6", ~ (1994)
- Dagan, I., "The PASCAL recognising textual entailment challenge", Machine Learning Challenges Workshop, pp.177~190 (2005), Springer
- Dostál, M., "Mathematics, Logic, and their Philosophies: Essays in Honour of Mohammad Ardeshir", pp.135~ (2020)
- Efron, B, "Breakthroughs in statistics" (1992), Springer
- Christina L. Gagné, "Is buttercup a kind of cup? Hyponymy and semantic transparency in compound words", Journal of Memory and Language, 113, pp.104110~ (2020), Elsevier BV[DOI]
- Geffet, Maayan, "The distributional inclusion hypotheses and lexical entailment", Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pp.107~114 (2005), Association for Computational Linguistics
- Geiger, Atticus, "Stress-testing neural models of natural language inference with multiplyquantified sentences", arXiv preprint arXiv:1810.13033, ~ (2018)
- Grefenstette, E., "Multi-step regression learning for compositional distributional semantics", Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) –Long Papers, pp.131~142 (2013), Association for Computational Linguistics
- Grefenstette, Edward, "Experimental support for a categorical compositional distributional model of meaning", Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pp.1394~1404 (2011), Association for Computational Linguistics
- Hearst, Marti A., "Automatic acquisition of hyponyms from large text corpora", COLING 1992 Volume 2: The 14th International Conference on Computational Linguistics (1992)
- Hedges, J., "A generalised quantifier theory of natural language in categorical compositional distributional semantics with bialgebras", Math. Struct. Comput. Sci, 29(6), pp.783~809 (2019)
- Herbelot, Aurélie, "Measuring semantic content in distributional vectors", Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, 2, pp.440~445 (2013), Association for Computational Linguistics
- Kartsaklis, Dimitri, "Resolving Lexical Ambiguity in Tensor Regression Models of Meaning", Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, 2, pp.212~217 (2014), Association for Computational Linguistics
- Kartsaklis, Dimitri, "Distributional inclusion hypothesis for tensor-based composition", Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp.2849~2860 (2016), The COLING 2016 Organizing Committee
- Kartsaklis, Dimitri, "A unified sentence space for categorical distributional-compositional semantics: Theory and experiments", Proceedings of COLING 2012: Posters, pp.549~558 (2012), The COLING 2012 Organizing Committee
- Kartsaklis, Dimitri, "Logic and Algebraic Structures in Quantum Computing, Lecture Notes in Logic", pp.199~222 (2016), Cambridge University Press
- Kiela, Douwe, "Exploiting image generality for lexical entailment detection", Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, 2, pp.119~124 (2015), Association for Computational Linguistics
- Kiros, Jamie, "Skip-thought vectors", Advances in neural information processing systems, pp.3294~3302 (2015)
- Kotlerman, L., "Directional distributional similarity for lexical inference", Nat. Lang. Eng, 16(4), pp.359~ (2010)
- Le, Matthew, "Inferring concept hierarchies from text corpora via hyperbolic embeddings", Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.3231~3241 (2019), Association for Computational Linguistics
- Lenci, Alessandro, "Identifying hypernyms in distributional semantic spaces", SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pp.75~79 (2012), Association for Computational Linguistics
- Lewis, Martha, "Compositional hyponymy with positive operators", Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pp.638~647 (2019), INCOMA Ltd
- Lewis, Martha, "Compositionality for Recursive Neural Networks", Journal of Applied Logics, 6, pp.709~724 (2019)
- Lewis, Martha, "Towards logical negation in compositional distributional semantics", Journal of Applied Logics, 7(5), pp.771~794 (2020)
- Lyons, John, "Introduction to theoretical linguistics, volume 510" (1968), Cambridge University Press
- MacCartney, Bill, "Natural logic for textual inference", Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pp.193~200 (2007), Association for Computational Linguistics
- Maillard, Jean, "Atype-driven tensorbased semantics for CCG", Proceedings of the EACL 2014 Workshop on Type Theory and Natural Language Semantics (TTNLS), pp.46~54 (2014), Association for Computational Linguistics
- Meyer, Francois, "Modelling Lexical Ambiguity with Density Matrices", Proceedings of the 24th Conference on Computational Natural Language Learning, pp.276~290 (2020), Association for Computational Linguistics
- Mikolov, Tomas, "Distributed representations of words and phrases and their compositionality", Advances in neural information processing systems, pp.3111~3119 (2013)
- Miller, George A., "WordNet : a lexical database for English", Communications of the ACM, 38(11), pp.39~41 (1995)
- Mitchell, Jeff, "Composition in distributional models of semantics", Cognitive Science, 34(8), pp.1388~1429 (2010)
- Moortgat, Michael, "A Frobenius algebraic analysis for parasitic gaps", Journal of Applied Logics, 7(5), pp.823~854 (2020)
- Moortgat, Michael, "Lexical and derivational meaning in vector-based models of relativisation", Proceedings of the 21st Amsterdam Colloquium, pp.55~64 (2017), ILLC, University of Amsterdam
- Murphy, MLynne, "Semantic relations and the lexicon: Antonymy, synonymy and other paradigms" (2003), Cambridge University Press
- Muskens, Reinhard, "Context update for lambdas and vectors", International Conference on Logical Aspects of Computational Linguistics, pp.247~254 (2016), Springer
- Nguyen, Kim Anh, "Hierarchical embeddings for hypernymy detection and directionality", Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp.233~243 (2017), Association for Computational Linguistics
- Nickel, Maximillian, "Poincaré embeddings for learning hierarchical representations", Advances in neural information processing systems, 30, pp.6338~6347 (2017)
- Nielsen, Michael A., "Quantum Computation and Quantum Information: 10th Anniversary Edition" (2010), Cambridge University Press
- Paperno, Denis, "A practical and linguistically-motivated approach to compositional distributional semantics", Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, 1, pp.90~99 (2014), Association for Computational Linguistics
- Paulsen, Vern, "Completely bounded maps and operator algebras", Cambridge Studies in Advanced Mathematics, 78 (2002), Cambridge University Press
- Pennington, Jeffrey, "GloVe: Global vectors for word representation", Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.1532~1543 (2014), Association for Computational Linguistics
- Piedeleu, Robin, "Open system categorical quantum semantics in natural language processing", 6th Conference on Algebra and Coalgebra in Computer Science (CALCO 2015) (2015), Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik
- Preller, Anne, "Semantic vector models and functional models for pregroup grammars", Journal of Logic, Language and Information, 20(4), pp.419~443 (2011)
- Reimers, Nils, "Sentence-BERT: Sentence embeddings using Siamese BERT-networks", Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (2019), Association for Computational Linguistics
- Rimell, Laura, "Distributional lexical entailment by topic coherence", Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pp.511~519 (2014), Association for Computational Linguistics
- Roller, Stephen, "Hearst patterns revisited: Automatic hypernym detection from large text corpora", Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, 2, pp.358~363 (2018), Association for Computational Linguistics
- Sadrzadeh, Mehrnoosh, "The Frobenius anatomy of word meanings I : subject and object relative pronouns", Journal of Logic and Computation, 23(6), pp.1293~1317 (2013)
- Sadrzadeh, Mehrnoosh, "Sentence entailment in compositional distributional semantics", Annals of Mathematics and Artificial Intelligence, 82(4), pp.189~218 (2018)
- Socher, Richard, "Recursive deep models for semantic compositionality over a sentiment treebank", Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp.1631~1642 (2013), Association for Computational Linguistics
- Vulii, Ivan, "Specialising word vectors for lexical entailment", Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, pp.1134~1145 (2018), Association for Computational Linguistics
- Weeds, Julie, "Characterising measures of lexical distributional similarity", COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, pp.1015~1021 (2004), COLING
- van de Wetering, John, "Ordering information on distributions", arXiv:1701.06924, ~ (2017)
- Winholds, Gis, "Categorical foundations for extended compositional distributional models of meaning", Master, pp.2015
- Winholds, Gis, "Representation Learning for Type-Driven Composition", Proceedings of the 24th Conference on ComputationalNatural Language Learning, pp.313~324 (2020), Association for Computational Linguistics
- Williams, Adina, "A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference", Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, pp.1112~1122 (2018), Association for Computational Linguistics
- Yanaka, Hitomi, "HELP: A Dataset for Identifying Shortcomings of Neural Models in Monotonicity Reasoning", Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), pp.250~255 (2019), Association for Computational Linguistics