Investigating syntactic transfer in second language learning of neural language models
2
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
Second language acquisition (SLA) research has extensively delved into cross-linguistic transfer, examining the impact of the linguistic structure of a native language on the acquisition of a second language. Such transfer effect can be either positive or negative, impeding the acquisition. In this paper, we employ transfer learning as a methodology for analyzing the encoding of grammatical structure in neural language models. This approach, transfer learning, involves pre-training the neural language model which is the Transformer-based language model, BabyRoberta, on Korean as the first language (L1). Afterward, we fine-tune the model with English as a second language (L2). Our task includes using the BLiMP test suite (Warstadt, 2020), broadly known as a benchmark for measuring the syntactic ability of neural language models. This allows us to provide insights into how neural language models represent abstract syntactic structures of English, incorporating the structural inductive biases acquired from Korean.
Abstract
Second language acquisition (SLA) research has extensively delved into cross-linguistic transfer, examining the impact of the linguistic structure of a native language on the acquisition of a second language. Such transfer effect can be either positive or negative, impeding the acquisition. In this paper, we employ transfer learning as a methodology for analyzing the encoding of grammatical structure in neural language models. This approach, transfer learning, involves pre-training the neural language model which is the Transformer-based language model, BabyRoberta, on Korean as the first language (L1). Afterward, we fine-tune the model with English as a second language (L2). Our task includes using the BLiMP test suite (Warstadt, 2020), broadly known as a benchmark for measuring the syntactic ability of neural language models. This allows us to provide insights into how neural language models represent abstract syntactic structures of English, incorporating the structural inductive biases acquired from Korean.
참고문헌 (21)
- Artetxe, M., "A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings", arXiv preprint arXiv:1805.06297, ~ (2018)
- Berzak, Y., "Reconstructing native language typology from foreign language usage", arXiv preprint arXiv:1404.6312, ~ (2014)
- Conneau, A., "Word translation without parallel data", arXiv preprint arXiv:1710.04087, ~ (2017)
- Derwing, T. M., "Teaching native speakers to listen to foreign-accented speech", Journal of Multilingual and Multicultural Development, 23(4), pp.245~259 (2002)
- Dulay, H. C., "Errors and strategies in child second language acquisition", TESOL Quarterly, 8(2), pp.129~136 (1974)
- Ettinger, A., "What BERT is not : Lessons from a new suite of psycholinguistic diagnostics for language models", Transactions of the Association for Computational Linguistics, 8, pp.34~48 (2020)
- Huebner, P. A., "BabyBERTa: Learning more grammar with small-scale child-directed language", Proceedings of the 25th conference on computational natural language learning, pp.624~646 (2021), Association for Computational Linguistics
- Kassner, N., "Negated and misprimed probes for pretrained language models : Birds can talk, but cannot fly", arXiv preprint arXiv:1911. 03343, ~ (2019)
- Liu, Y., "RoBERTa: A robustly optimized BERT pretraining approach", arXiv preprint arXiv:1907.11692, 364, ~ (2019)
- Jarvis, S., "Crosslinguistic influence in language and cognition" (2008), Routledge
- Papadimitriou, I., "Learning music helps you read : Using transfer to study linguistic structure in language models", arXiv preprint arXiv:2004.14601, ~ (2020)
- Ringbom, H, "Cross-linguistic similarity in foreign language learning (Vol. 21)" (2007), Multilingual Matters
- Ruder, S., "A survey of cross-lingual word embedding models", Journal of Artificial Intelligence Research, 65, pp.569~631 (2019)
- Sprouse, J., "Experimental syntax and island effects", pp.1~20 (2013), Cambridge University Press
- Schwab, J., "On the acquisition of polarity items : 11-to 12-year-olds' comprehension of German NPIs and PPIs", Journal of Psycholinguistic Research, 50(6), pp.1487~1509 (2021)
- Tieu, L., "Logic and grammar in child language: How children acquire the semantics of polarity sensitivity", Doctor, pp.2013
- Van der Wal, S., "Negative polarity items and negation: Tandem acquisition", Doctor, pp.1996
- Warstadt, A., "BLiMP : The benchmark of linguistic minimal pairs for English", Transactions of the Association for Computational Linguistics, 8, pp.377~392 (2020)
- Wu, S., "Emerging cross-lingual structure in pretrained language models", arXiv preprint arXiv:1911.01464, ~ (2019)
- Wu, S., "Beto, bentz, becas : The surprising cross-lingual effectiveness of BERT", arXiv preprint arXiv:1904.09077, ~ (2019)
- Yadavalli, A., "SLABERT talk pretty one day : Modeling second language acquisition with BERT", arXiv preprint arXiv:2305.19589, ~ (2023)