Can an L2-neural LM Generalize Filler-gap Dependency?
2
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Recent studies have shown that recurrent neural language models (LMs) can understand sentences involving filler-gap dependency (Chowdhury & Zamparelli, 2018; Wilcox et al., 2018, 2019). However, their behavior does not encode the underlying constraints that govern filler-gap acceptability. In this vein, significant issues remain about the extent to which LMs acquire specific linguistic constructions and whether these models recognize an abstract property of syntax in their representations. In this paper, following the lead of Bhattacharya and van Schijndel (2020), we further test whether the L2 neural LM can learn abstract syntactic constraints that have been claimed to govern the behavior of filler-gap constructions. To see this, we implement the L2 neural LM trained on the L2 corpus of English textbooks published in Korea for the last two decades, and then we test the representational overlap between disparate filler-gap constructions based on the syntactic priming paradigm. Unlike the previous studies of L1-neural LMs, we could not find sufficient evidence showing that the L2 neural LM learns a general representation of the existence of filler-gap dependency and the shared underlying constraints.
Abstract
Recent studies have shown that recurrent neural language models (LMs) can understand sentences involving filler-gap dependency (Chowdhury & Zamparelli, 2018; Wilcox et al., 2018, 2019). However, their behavior does not encode the underlying constraints that govern filler-gap acceptability. In this vein, significant issues remain about the extent to which LMs acquire specific linguistic constructions and whether these models recognize an abstract property of syntax in their representations. In this paper, following the lead of Bhattacharya and van Schijndel (2020), we further test whether the L2 neural LM can learn abstract syntactic constraints that have been claimed to govern the behavior of filler-gap constructions. To see this, we implement the L2 neural LM trained on the L2 corpus of English textbooks published in Korea for the last two decades, and then we test the representational overlap between disparate filler-gap constructions based on the syntactic priming paradigm. Unlike the previous studies of L1-neural LMs, we could not find sufficient evidence showing that the L2 neural LM learns a general representation of the existence of filler-gap dependency and the shared underlying constraints.
참고문헌 (29)
- Baroni, M, "On the proper role of linguistically-oriented deep net analysis in linguistic theorizing", arXiv preprint arXiv:2106.08694, ~ (2021)
- Bhattacharya, D., "Filler-gaps that neural networks fail to generalize", Proceedings of the 24th Conference on Computational Natural Language Learning, pp.486~495 (2020)
- Bock, J. K, "Syntactic persistence in language production", Cognitive Psychology, 18(3), pp.355~387 (1986)
- Chang, F., "Becoming syntactic", Psychological Review, 113(2), pp.234~ (2006)
- Chaves, R. P., "What don’t RNN language models learn about filler-gap dependencies?", Proceedings of the Society for Computation in Linguistics, 3(1), pp.20~30 (2020)
- 최선주, "An L2 Neural Language Model of Adaptation to Dative Alternation in English", 현대영미어문학, 40(1), pp.143~159 (2022), 현대영미어문학회
- 최선주, "Syntactic Priming by L2 LSTM Language Models", 언어연구, 37(4), pp.475~489 (2022), 한국현대언어학회[DOI]
- 최선주, "An L2 Neural Language Model of Adaptation", 영어학, 22, pp.547~562 (2022), 한국영어학회[DOI]
- 최선주, "How are Korean Neural Language Models ‘surprised’ Layerwisely?", 언어과학, 28(4), pp.301~317 (2021), 한국언어과학회[DOI]
- Chomsky, N., "A Festschrift for Morris Halle", pp.232~286 (1973), Holt, Rinehart & Winston
- Chomsky, N., "Barriers" (1986), MIT Press
- Chowdhury, S. A., "RNN simulations of grammaticality judgments on long-distance dependencies", Proceedings of the 27th International Conference on Computational Linguistics, pp.133~144 (2018)
- Ettinger, A, "What BERT is not : Lessons from a new suite of psycholinguistic diagnostics for language models", Transactions of the Association for Computational Linguistics, 8, pp.34~48 (2020)
- Futrell, R., "Neural language models as psycholinguistic subjects : Representations of syntactic state", arXiv preprint arXiv:1903.03260, ~ (2019)
- Gulordava, K., "Colorless green recurrent networks dream hierarchically", arXiv preprint arXiv:1803.11138, ~ (2018)
- Kaschak, M. P., "Long-term cumulative structural priming persists for (at least) one week", Memory & Cognition, 39(3), pp.381~388 (2011)
- 김유희, "The Ability of L2 LSTM Language Models to Learn the Filler-Gap Dependency", 한국컴퓨터정보학회논문지, 25(11), pp.27~40 (2020), 한국컴퓨터정보학회[DOI]
- Kodner, J., "Overestimation of syntactic representation in neural language models", arXiv preprint arXiv:2004.05067, ~ (2020)
- useful : Effects of constituency and dependency structure in recursive LSTMs, "arXiv preprint arXiv:2005.00019", 2020
- Linzen, T., "Assessing the ability of LSTMs to learn syntax-sensitive dependencies", Transactions of the Association for Computational Linguistics, 4, pp.521~535 (2016)
- Pesetsky, D., "Wh-in-situ: Movement and unselective binding" (1987), The Representation of (In)Definiteness
- Prasad, G., "Using priming to uncover the organization of syntactic representations in neural language models", arXiv preprint arXiv:1909.10579, ~ (2019)
- Ross, J. R., "Constraints on Variables in Syntax", ERIC, ED016965, ~ (1967)
- Sinclair, A., "Syntactic persistence in language models : Priming as a window into abstract language representations", arXiv preprint arXiv:2109.14989, ~ (2022)
- Tenney, I., "BERT rediscovers the classical NLP pipeline", arXiv preprint arXiv:1905.05950, ~ (2019)
- Van Schijndel, M., "A neural model of adaptation in reading", arXiv preprint arXiv:1808.09930, ~ (2018)
- Van Schijndel, M., "Quantity doesn’t buy quality syntax with neural language models", arXiv preprint arXiv:1909.00111, ~ (2019)
- Wilcox, E., "Hierarchical representation in neural language models : Suppression and recovery of expectations", arXiv preprint arXiv:1906.04068, ~ (2019)
- Wilcox, E., "What do RNN language models learn about filler-gap dependencies?", arXiv preprint arXiv:1809.00042, ~ (2018)