Syntactic Priming by L2 LSTM Language Models
2
조회
0
다운로드
6
인용
0
참고문헌
키워드
초록
Neural(-network) language models (LMs) have recently been successful in performing the tasks that require sensitivity to syntactic structure. We provide further evidence for their sensitivity to syntactic structure by showing that compared to adding a non-adaptive counterpart to it, adding an adaptation-as-priming paradigm to L2 LSTM LMs improves their ability to track abstract structure. By applying a gradient similarity metric between structures, this mechanism allows us to reconstruct the organization of the L2 LMs’ syntactic representational space. In so doing, we discover that sentences with a particular type of relative clauses behave in a similar fashion to other sentences with the same type of relative clauses in the L2 LMs’ representation space, in keeping with the recent studies of L1 LM adaptation. We also demonstrate that the similarity between given sentences is not affected by specific words in sentences. Our results show that the L2 LMs have the ability to track abstract structural properties of sentences, just as L1 LMs do.
Abstract
Neural(-network) language models (LMs) have recently been successful in performing the tasks that require sensitivity to syntactic structure. We provide further evidence for their sensitivity to syntactic structure by showing that compared to adding a non-adaptive counterpart to it, adding an adaptation-as-priming paradigm to L2 LSTM LMs improves their ability to track abstract structure. By applying a gradient similarity metric between structures, this mechanism allows us to reconstruct the organization of the L2 LMs’ syntactic representational space. In so doing, we discover that sentences with a particular type of relative clauses behave in a similar fashion to other sentences with the same type of relative clauses in the L2 LMs’ representation space, in keeping with the recent studies of L1 LM adaptation. We also demonstrate that the similarity between given sentences is not affected by specific words in sentences. Our results show that the L2 LMs have the ability to track abstract structural properties of sentences, just as L1 LMs do.
참고문헌 (14)
- Bock, J. K, "Syntactic persistence in language production", Cognitive Psychology, 18, pp.355~387 (1986)
- Chang, F., "Becoming syntactic", Psychological Review, 113, pp.234~272 (2006)
- Dubey, A., "Integrating syntactic priming into an incremental probabilistic parser, with an application to psycholinguistic modeling", Proceedings of the 21st international Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pp.417~424 (2006)
- Gulordava, K., "Colorless green recurrent networks dream hierarchically", Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.1195~1205 (2018)
- Kaschak, M. P., "Long-term cumulative structural priming persists for(at least)one week", Memory and Cognition, 39, pp.381~388 (2011)
- Kim, E., "Probing sentence embeddings in L2 learners’ LSTM neural language models using adaptation learning", Ms, pp.2022
- Kuhn, R., "A cache-based natural language model for speech recognition", IEEE Transactions on Pattern Analysis and Machine Intelligence, 12, pp.570~583 (1990)
- Linzen, T., "Assessing the ability of LSTMs to learn syntax-sensitive dependencies", Transactions of the Association for Computational Linguistics, 4, pp.521~535 (2016)
- Marvin, R., "Targeted syntactic evaluation of language models", Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.1192~1202 (2018)
- Pinheiro, J., "Mixed-effects Models in S and S-PLUS" (2000), Springer Science & Business Media
- Prasad, G., "Using priming to uncover the organization of syntactic representations in neural language models", Proceedings of the 23rd Conference on Computational Natural Language Learning, pp.66~76 (2019)
- Roland, D., "Frequency of basic English grammatical structures : A corpus analysis", Journal of Memory and Language, 57, pp.348~379 (2007)
- Van Schijndel, M., "A neural model of adaptation in reading", Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.4704~4710 (2018)
- Van Schijndel, M., "Quantity doesn’t buy quality syntax with neural language models", Proceedings of the 2019 conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp.5831~5837 (2019)