The Ability of L2 LSTM Language Models to Learn the Filler-Gap Dependency
2
조회
0
다운로드
6
인용
0
참고문헌
키워드
초록
In this paper, we investigate the correlation between the amount of English sentences that Korean English learners (L2ers) are exposed to and their sentence processing patterns by examining what Long Short-Term Memory (LSTM) language models (LMs) can learn about implicit syntactic relationship: that is, the filler–gap dependency. The filler–gap dependency refers to a relationship between a (wh-)filler, which is a wh-phrase like ‘what’ or ‘who’ overtly in clause-peripheral position, and its gap in clause-internal position, which is an invisible, empty syntactic position to be filled by the (wh-)filler for proper interpretation. Here to implement L2ers’ English learning, we build LSTM LMs that in turn learn a subset of the known restrictions on the filler-gap dependency from English sentences in the L2 corpus that L2ers can potentially encounter in their English learning. Examining LSTM LMs’ behaviors on controlled sentences designed with the filler-gap dependency, we show the characteristics of L2ers' sentence processing using the information-theoretic metric of surprisal that quantifies violations of the filler-gap dependency or wh-licensing interaction effects. Furthermore, comparing L2ers’ LMs with native speakers’ LM in light of processing the filler-gap dependency, we not only note that in their sentence processing both L2ers’ LM and native speakers’ LM can track abstract syntactic structures involved in the filler-gap dependency, but also show using linear mixed-effects regression models that there exist significant differences between them in processing such a dependency.
Abstract
In this paper, we investigate the correlation between the amount of English sentences that Korean English learners (L2ers) are exposed to and their sentence processing patterns by examining what Long Short-Term Memory (LSTM) language models (LMs) can learn about implicit syntactic relationship: that is, the filler–gap dependency. The filler–gap dependency refers to a relationship between a (wh-)filler, which is a wh-phrase like ‘what’ or ‘who’ overtly in clause-peripheral position, and its gap in clause-internal position, which is an invisible, empty syntactic position to be filled by the (wh-)filler for proper interpretation. Here to implement L2ers’ English learning, we build LSTM LMs that in turn learn a subset of the known restrictions on the filler-gap dependency from English sentences in the L2 corpus that L2ers can potentially encounter in their English learning. Examining LSTM LMs’ behaviors on controlled sentences designed with the filler-gap dependency, we show the characteristics of L2ers' sentence processing using the information-theoretic metric of surprisal that quantifies violations of the filler-gap dependency or wh-licensing interaction effects. Furthermore, comparing L2ers’ LMs with native speakers’ LM in light of processing the filler-gap dependency, we not only note that in their sentence processing both L2ers’ LM and native speakers’ LM can track abstract syntactic structures involved in the filler-gap dependency, but also show using linear mixed-effects regression models that there exist significant differences between them in processing such a dependency.
참고문헌 (16)
- J. L. Elman, "Distributed representations, simple recurrent networks, and grammatical structure", Machine learning, 7(2-3), pp.195~225 (1991)
- Y. Goldberg, "Neural network methods for natural language processing", Synthesis lectures on Human language Technologies, 10(1), pp.1~309 (2017)
- S. Hochreiter, "Long short-term memory", Neural Computation, 9(8), pp.1735~1780 (1997)
- 김유희, "Sentence Comprehension with an LSTM Language Model", 디지털콘텐츠학회논문지, 19(12), pp.2393~2401 (2018), 한국디지털콘텐츠학회[DOI]
- T. Linzen, "Assessing the ability of LSTMs to learn syntax-sensitive dependencies", Transactions of the Association for Computational Linguistics, 4, pp.521~535 (2016)
- K. Gulordava, "Colorless green recurrent networks dream hierarchically", NAACL-HLT, pp.1195~1205 (2018)
- E. Wilcox, "What do RNN Language Models Learn about Filler-Gap Dependencies?", ACL Anthology, Proceedings of the 2018 EMNLP Workshop Blackbox NLP: Analyzing and Interpreting Neural Networks for NLP, pp.211~221 (2019)
- A. Kuncoro, "LSTMs can learn syntax-sensitive dependencies well, but modeling structure makes them better", Computational Linguistics, 1, pp.1426~1436 (2018)
- K. Tran, "The importance of being recurrent for modeling hierarchical structure", Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (2018)
- J. Hale, "Uncertainty about the rest of the sentence", Cognitive Science, 30(4), pp.609~642 (2006)
- J. Hale, "A probabilistic Earley parser as a psycholinguistic model", Proceedings of the Second meeting of the North American Chapter of the Association for Computational Linguistic and Language Technologies, pp.1~8 (2001)
- R. Levy, "Expectation-based syntactic comprehension", Cognition, 106(3), pp.1126~1177 (2008)
- R. H. Baayen, "Mixed-effects modeling with crossed random effects for subjects and items", Journal of memory and language, 59(4), pp.390~412 (2008)
- 김유희, "On Korean English L2ers’ processing of wh-filler-gap dependencies: An ERP study", 언어와 정보, 21(3), pp.1~24 (2017), 한국언어정보학회
- 김유희, "딥러닝 기반 기사단위 및 문단 단위별 분류", 한국컴퓨터정보학회논문지, 23(11), pp.31~41 (2018), 한국컴퓨터정보학회[DOI]
- 김유희, "한국어 단어 이해에 관한 비지도 학습 기반 언어 모델링", 한국컴퓨터정보학회논문지, 24(11), pp.41~49 (2019), 한국컴퓨터정보학회[DOI]