Deep learning can contrast the minimal pairs of syntactic data
1
조회
0
다운로드
8
인용
0
참고문헌
키워드
초록
The present work aims to assess the feasibility of using deep learning as a useful tool to investigate syntactic phenomena. To this end, the present study concerns three research questions: (i) whether deep learning can detect syntactically inappropriate constructions, (ii) whether deep learning’s acceptability judgments are accountable, and (iii) whether deep learning’s aspects of acceptability judgments are similar to human judgments. As a proxy for a deep learning language model, this study chooses BERT. The current paper comprises syntactically contrasted pairs of English sentences which come from the three test suites already available. The first one is 196 grammatical –ungrammatical minimal pairs from DeKeyser (2000). The second one is examples in four published syntax textbooks excerpted from Warstadt et al. (2019). The last one is extracted from Sprouse et al. (2013), which collects the examples reported in a theoretical linguistics journal, Linguistic Inquiry. The BERT models, base BERT and large BERT, are assessed by judging acceptability of items in the test suites with an evaluation metric, surprisal, which is used to measure how ‘surprised’ a model is when encountering a word in a sequence of words, i.e., a sentence. The results are analyzed in the two frameworks: directionality and repulsion. The results of directionality reveals that the two versions of BERT are overall competent at distinguishing ungrammatical sentences from grammatical ones. The statistical results of both repulsion and directionality also reveal that the two variants of BERT do not differ significantly. Regarding repulsion, correct judgments and incorrect ones are significantly different. Additionally, the repulsion of the first test suite, which is excerpted from the items for testing learners’ grammaticality judgments, is higher than the other test suites, which are excerpted from the syntax textbooks and published literature. This study compares BERT’s acceptability judgments with magnitude estimation results reported in Sprouse et al. (2013) in order to examine if deep learning’s syntactic knowledge is akin to human knowledge. The error analyses on incorrectly judged items reveal that there are some syntactic constructions that the two BERTs have trouble learning, which indicates that BERT’s acceptability judgments are distributed not randomly.
Abstract
The present work aims to assess the feasibility of using deep learning as a useful tool to investigate syntactic phenomena. To this end, the present study concerns three research questions: (i) whether deep learning can detect syntactically inappropriate constructions, (ii) whether deep learning’s acceptability judgments are accountable, and (iii) whether deep learning’s aspects of acceptability judgments are similar to human judgments. As a proxy for a deep learning language model, this study chooses BERT. The current paper comprises syntactically contrasted pairs of English sentences which come from the three test suites already available. The first one is 196 grammatical –ungrammatical minimal pairs from DeKeyser (2000). The second one is examples in four published syntax textbooks excerpted from Warstadt et al. (2019). The last one is extracted from Sprouse et al. (2013), which collects the examples reported in a theoretical linguistics journal, Linguistic Inquiry. The BERT models, base BERT and large BERT, are assessed by judging acceptability of items in the test suites with an evaluation metric, surprisal, which is used to measure how ‘surprised’ a model is when encountering a word in a sequence of words, i.e., a sentence. The results are analyzed in the two frameworks: directionality and repulsion. The results of directionality reveals that the two versions of BERT are overall competent at distinguishing ungrammatical sentences from grammatical ones. The statistical results of both repulsion and directionality also reveal that the two variants of BERT do not differ significantly. Regarding repulsion, correct judgments and incorrect ones are significantly different. Additionally, the repulsion of the first test suite, which is excerpted from the items for testing learners’ grammaticality judgments, is higher than the other test suites, which are excerpted from the syntax textbooks and published literature. This study compares BERT’s acceptability judgments with magnitude estimation results reported in Sprouse et al. (2013) in order to examine if deep learning’s syntactic knowledge is akin to human knowledge. The error analyses on incorrectly judged items reveal that there are some syntactic constructions that the two BERTs have trouble learning, which indicates that BERT’s acceptability judgments are distributed not randomly.
참고문헌 (45)
- Adger, David, "Core syntax : A minimalist approach" (2003), Oxford University Press
- 안희돈, "한국어 수용성 판단에서의 형식적 방법과 비형식적 방법의 비교 - 이산형 측정 자료의 경우", 한말연구, (51), pp.57~86 (2019), 한말연구학회[DOI]
- Alain, Guillaume, "Understanding intermediate layers using linear classifier probes", arXiv preprint arXiv:1610.01644, ~ (2016)
- Bock, Kathryn, "Broken agreement", Cognitive Psychology, 23(1), pp.45~93 (1991)
- Carnie, Andrew, "Syntax : A generative introduction" (2013), John Wiley and Sons
- Cecchetto, Carlo, "A comparative analysis of left and right dislocation in Romance", Studia Linguistica, 53(1), pp.40~67 (1999)
- Chicco, Davide, "The advantages of the Matthews correlation coefficient(MCC)over F1 score and accuracy in binary classification evaluation", BMC Genomics, 21(1), pp.6~ (2020)
- 조용준, "한국어 수용성 판단에서의 형식적 방법과 비형식적 방법의 비교(2)", 언어학 연구, (51), pp.129~160 (2019), 한국중원언어학회[DOI]
- Collins, Chris, "Quotative inversion", Natural Language and Linguistic Theory, 15(1), pp.1~41 (1997)
- Culicover, Peter W., "Control is not movement", Linguistic Inquiry, 32(3), pp.493~512 (2001)
- Da Costa, "Assessing the ability of Transformer-based Neural Models to represent structurally unbounded dependencies", Proceedings of the Society for Computation in Linguistics, 3(1), pp.189~198 (2020)
- DeKeyser, Robert M, "The robustness of critical period effects in second language acquisition", Studies in Second Language Acquisition, 22(4), pp.499~533 (2000)
- Dikken, Marcel den, "From hell to polarity: “Aggressively non-D-linked” wh-phrases as polarity items", Linguistic Inquiry, 33(1), pp.31~61 (2002)
- Devlin, Jacob, "Bert : Pre-training of deep bidirectional transformers for language understanding", arXiv preprint arXiv:1810.04805, ~ (2018)
- Goldberg, Yoav, "Assessing BERT’s syntactic abilities", arXiv preprint arXiv:1901.05287, ~ (2019)
- Hale, John, "A probabilistic Earley parser as a psycholinguistic model", Second Meeting of the North American Chapter of the Association for Computational Linguistics, 2, pp.159~166 (2001)
- Hornstein, Norbert, "A very short note on existential constructions", Linguistic Inquiry, 38(2), pp.410~411 (2007)
- Johnson, Jacqueline S., "Critical period effects in second language learning : The influence of maturational state on the acquisition of English as a second language", Cognitive Psychology, 21(1), pp.60~99 (1989)
- Johnson, Kyle, "Gapping is not(VP-)ellipsis", Linguistic Inquiry, 40(2), pp.289~328 (2009)
- Keller, Frank, "Gradience in grammar : Experimental and computational aspects of degrees of grammaticality", PhD, pp.2000
- 김유희, "L2ers’ predictions of syntactic structure and reaction times during sentence processing", 언어연구, 37, pp.189~218 (2020), 언어정보연구소[DOI]
- Lakoff, Robin, "Acceptability in language", pp.73~86 (1977), Mouton Publishers
- Levy, Roger, "Expectation-based syntactic comprehension", Cognition, 106(3), pp.1126~1177 (2008)
- Linzen, Tal, "What can linguistics and deep learning contribute to each other?", arXiv preprint arXiv:1809.04179, ~ (2018)
- Linzen, Tal, "The reliability of acceptability judgments across languages", Glossa: A Journal of General Linguistics, 3(1) (2018)
- Marvin, Rebecca, "Targeted syntactic evaluation of language models", arXiv preprint arXiv:1808.09031, ~ (2018)
- Matthews, Brian W, "Comparison of the predicted and observed secondary structure of T4phage lysozyme", Biochimica et Biophysica Acta(BBA)-Protein Structure, 405(2), pp.442~451 (1975)
- McCoy, R. Thomas, "Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference", arXiv preprint arXiv:1902.01007, ~ (2019)
- Miller, Jim, "An introduction to English syntax" (2002), Edinburgh University Press
- Moulton, Keir, "Focus on cataphora: Experiments in context", Linguistic Inquiry, 49(1), pp.151~168 (2018)
- Phillips, Colin, "Linear order and constituency", Linguistic Inquiry, 34(1), pp.37~90 (2003)
- Song, Sanghoun, "Modeling information structure in a cross-linguistic perspective" (2017), Language Science Press
- 송상헌, "What do you mean by contrast in syntax?", 언어연구, 34(3), pp.387~426 (2017), 언어정보연구소[DOI]
- Sorace, Antonella, "Gradience in linguistic data", Lingua, 115(11), pp.1497~1524 (2005)
- Sportiche, Dominique, "An introduction to syntactic analysis and theory" (2013), John Wiley and Sons
- Sprouse, Jon, "A comparison of informal and formal acceptability judgments using a random sample from Linguistic Inquiry 2001–2010", Lingua, 134, pp.219~248 (2013)
- Takano, Yuji, "How antisymmetric is syntax?", Linguistic Inquiry, 34(3), pp.516~526 (2003)
- Taylor, Wilson L, "“Cloze procedure”: A new tool for measuring readability", Journalism Quarterly, 30(4), pp.415~433 (1953)
- Vaswani, Ashish, "Attention is all you need", arXiv preprint arXiv:1706.03762, ~ (2017)
- Wang, Alex, "Superglue : A stickier benchmark for general-purpose language understanding systems", arXiv preprint arXiv:1905.00537, ~ (2019)
- Wang, Alex, "Glue : A multi-task benchmark and analysis platform for natural language understanding", arXiv preprint arXiv:1804.07461, ~ (2018)
- Warstadt, Alex, "Neural network acceptability judgments", Transactions of the Association for Computational Linguistics, 7, pp.625~641 (2019)
- Warstadt, Alex, "BLiMP : The benchmark of linguistic minimal pairs for English", Transactions of the Association for Computational Linguistics, 8, pp.377~392 (2020)
- Wilcox, Ethan, "Hierarchical representation in neural language models: Suppression and recovery of expectations", arXiv preprint arXiv:1906.04068, ~ (2019)
- Zhu, Yukun, "Aligning books and movies : Towards story-like visual explanations by watching movies and reading books", Proceedings of the IEEE International Conference on Computer Vision, pp.19~27 (2015)