BERT Learns More than Word Frequency Information: A Case Study of Do-Be Constructions
0
조회
0
다운로드
4
인용
0
참고문헌
키워드
초록
This study aims to understand BERT’s linguistic ability using naturally occurring data. In particular, the study collected marginal language data, such as what we do is create Frankenstein, which is referred to as a Do-Be construction (DBC) (Flickinger & Wasow, 2013). Using web corpora, the study first collected 17,737 instances of the DBC across text genres and English dialects. The corpus analysis supports the idea that DBC is a computationally challenging phenomenon for data-driven language systems due to its statistical sparsity and linguistic complexity. With manual annotations of DBCs, the study designed two computational prediction tasks: subject―verb agreement and synonym substitution tasks, based on the introspective judgment of linguists. The study found that BERT is hugely sensitive to linguistic acceptability of grammatical forms and felicitous words in the prediction tasks, even though the target phenomenon is rarely observed in corpus data. These results show that the neural language model, BERT, can learn abstract linguistic properties beyond surface frequency information.
Abstract
This study aims to understand BERT’s linguistic ability using naturally occurring data. In particular, the study collected marginal language data, such as what we do is create Frankenstein, which is referred to as a Do-Be construction (DBC) (Flickinger & Wasow, 2013). Using web corpora, the study first collected 17,737 instances of the DBC across text genres and English dialects. The corpus analysis supports the idea that DBC is a computationally challenging phenomenon for data-driven language systems due to its statistical sparsity and linguistic complexity. With manual annotations of DBCs, the study designed two computational prediction tasks: subject―verb agreement and synonym substitution tasks, based on the introspective judgment of linguists. The study found that BERT is hugely sensitive to linguistic acceptability of grammatical forms and felicitous words in the prediction tasks, even though the target phenomenon is rarely observed in corpus data. These results show that the neural language model, BERT, can learn abstract linguistic properties beyond surface frequency information.
참고문헌 (26)
- Bock, Kathryn, "Broken agreement", Cognitive psychology, 23(1), pp.45~93 (1991)
- Brown, Lawrence D., "Interval estimation for a binomial proportion", Statistical science, 16(2), pp.101~133 (2001)
- Brown, Peter F., "Class-based n-gram models of natural language", Computational linguistics, 18(4), pp.467~480 (1992)
- Butt, Miriam, "Complex predicates in cross-linguistic perspective", pp.48~78 (2010)
- Chaves, Rui P., "Look at that! BERT can be easily distracted from paying attention to morphosyntax", Proceedings of the Society for Computation in Linguistics, 4(1), pp.28~38 (2021)
- Davies, Mark, "The 385+ million word Corpus of Contemporary American English(1990–2008+) : Design, architecture, and linguistic insights", International journal of corpus linguistics, 14(2), pp.159~190 (2009)
- Davies, Mark, "Expanding horizons in the study of World Englishes with the 1.9 billion word Global Web-based English Corpus (GloWbE)", English World-Wide, 36(1), pp.1~28 (2015)
- Davies, Mark, "The TV and Movies corpora: Design, construction, and use", International Journal of Corpus Linguistics, 26(1), pp.10~37 (2021)
- Devlin, Jacob, "Bert: Pre-training of deep bidirectional transformers for language understanding", Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, pp.4171~4186 (2019)
- Dillon, Brian, "Contrasting intrusion profiles for agreement and anaphora: Experimental and modeling evidence", Journal of Memory and Language, 69(2), pp.85~103 (2013)
- Flickinger, Dan, "On Building a More Efficient Grammar by Exploiting Types", Natural Language Engineering, 6(1), pp.15~28 (2000)
- Flickinger, Dan, "The core and the periphery: Data-driven perspectives on syntax inspired", pp.35~63 (2013), Sag
- Goldberg, Yoav, "Assessing BERT's syntactic abilities", arXiv preprint arXiv:1901.05287 (2019)
- Hale, John, "A probabilistic Earley parser as a psycholinguistic model", Second meeting of the north american chapter of the association for computational linguistics (2001)
- LeCun, Yan, "Deep learning", nature, 521(7553), pp.436~444 (2015)
- Linzen, Tal, "Assessing the ability of LSTMs to learn syntax-sensitive dependencies", Transactions of the Association for Computational Linguistics, 4, pp.521~535 (2016)
- Marvin, Rebecca, "Targeted syntactic evaluation of language models", Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.1192~1202 (2018)
- Rogers, Anna, "A primer in bertology:What we know about how bert works", Transactions of the Association for Computational Linguistics, 8, pp.842~866 (2020)
- Srivastava, "Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models", arXiv preprint arXiv:1902.04094 (2022)
- Wallis, S., "Statistics in corpus linguistics research: A new approach" (2020), Routledge
- Wagers, M. W., "Agreement attraction in comprehension: Representations and processes", Journal of memory and language, 61(2), pp.206~237 (2009)
- Wang, Alex, "Bert has a mouth, and it must speak: Bert as a markov random field language model", arXiv preprint arXiv:1902.04094, pp.04094~ (2019)
- Wasow, Thomas, "Expecting the unexpected: Exceptions in grammar", pp.75~96 (2011)
- Wasow, Thomas., "Explicit and implicit prosody in sentence processing", pp.133~158 (2015)
- Wei, Jason, "Frequency effects on syntactic rule learning in transformers", Proceedings of the 2021Conference on Empirical Methods in Natural Language Processing, pp.932~948 (2021)
- Wolf, "Transformers: State-of-the-art natural language processing", Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp.38~45 (2020)