What Can BERT Do for English Linguistics?: A Case Study of Examining It-cleft Constructions
0
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
This study used the machine learning technique known as Bidirectional Encoder Representations from Transformers (BERT) to understand the pronominal distribution in it-cleft constructions. While language models (LMs) have successfully been used to examine and verify linguistic representations, most studies have focused on the general phenomenon of subject-verb agreement. As such, this study focused on the particular construction known as it-cleft, in order to consolidate the LM’s role as a linguistic platform. In the experiment using BERT, the surprisal of each case (i.e., nominative and accusative) is compared. The investigation successfully replicated the findings of previous studies, specifically by capturing the use of the accusative form, inhibition of the first-person pronominal, and collocate feature of the first-person pronominal with the complementizer who. The results from the LM also deliver further contribution to the study of it-cleft, involving the between number and the relationship between the pronominal and syntactic role. Thus, the usage of BERT both revealed new research questions while verifying evidence from previous studies on the it-cleft pronominal. This demonstrates that BERT can correctly represent particular constructions and broadens the possibilities of LMs as a linguistic platform.
Abstract
This study used the machine learning technique known as Bidirectional Encoder Representations from Transformers (BERT) to understand the pronominal distribution in it-cleft constructions. While language models (LMs) have successfully been used to examine and verify linguistic representations, most studies have focused on the general phenomenon of subject-verb agreement. As such, this study focused on the particular construction known as it-cleft, in order to consolidate the LM’s role as a linguistic platform. In the experiment using BERT, the surprisal of each case (i.e., nominative and accusative) is compared. The investigation successfully replicated the findings of previous studies, specifically by capturing the use of the accusative form, inhibition of the first-person pronominal, and collocate feature of the first-person pronominal with the complementizer who. The results from the LM also deliver further contribution to the study of it-cleft, involving the between number and the relationship between the pronominal and syntactic role. Thus, the usage of BERT both revealed new research questions while verifying evidence from previous studies on the it-cleft pronominal. This demonstrates that BERT can correctly represent particular constructions and broadens the possibilities of LMs as a linguistic platform.
참고문헌 (47)
- Akmajian, A, "On Deriving Cleft Sentences from Pseudo-cleft Sentences", Linguistic Inquiry, 1, pp.49~168 (1970)
- Ball, C. N., "The Historical Development of the ‘It’-cleft", Doctor, pp.1991
- Bevacqua, L., "Form Variation of Pronominal It-clefts in Written English", Linguistics Vanguard, 6(1), pp.1~15 (2020)
- Biber, D., "Longman Grammar of Spoken and Written English" (1999), Longman
- "The Corpus of Contemporary American English (COCA)" (2008)[URL]
- "The Corpus of Historical American English (COHA)" (2010)[URL]
- Devlin, J., "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", arXiv preprint arXiv:1810.04805, ~ (2019)
- Ettinger, A, "What BERT is Not : Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models", Transactions of the Association for Computational Linguistics, 8, pp.34~48 (2020)
- Futrell, R., "Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State", arXiv preprint arXiv:1903.03260, ~ (2019)
- Geluykens, R, "Five Types of Clefting in English Discourse", Linguistics, 26(5), pp.823~841 (1988)
- Glockner, M., "Breaking NLI Systems with Sentences that Require Simple Lexical Inferences", arXiv preprint arXiv:1805.02266, ~ (2018)
- Goldberg, Y, "Assessing BERT’s Syntactic Abilities", arXiv preprint arXiv:1901.05287, ~ (2019)
- Gulordava, K., "Colorless Green Recurrent Networks Dream Hierarchically", arXiv preprint arXiv:1803.11138, ~ (2018)
- Gundel, J. K, "Where Do Cleft Sentences Come from?", Language, 53(3), pp.543~559 (1977)
- Han, C. H., "Syntax and Semantics of It-clefts: A Tree Adjoining Grammar Analysis", Journal of Semantics, 25(4), pp.345~380 (2008)
- Hale, J., "A Probabilistic Earley Parser as a Psycholinguistic Model", Second Meeting of the North American Chapter of the Association for Computational Linguistics (2001)
- Huddleston, R., "The Cambridge Grammar of the English Language" (2002), Cambridge University Press
- Hewitt, J., "A Structural Probe for Finding Syntax in Word Representations", Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies, 1, pp.4129~4138 (2019)
- Jawahar, G., "What Does BERT Learn about the Sructure of Language?", ACL 2019-57th Annual Meeting of the Association for Computational Linguistics (2019)
- Jumelet, J., "Do Language Models Understand Anything? On the Ability of LSTMs to Understand Negative Polarity Items", arXiv preprint arXiv:1808.10627, ~ (2018)
- 김종복, "On the Syntax of the It-Cleft Construction: A Construction-based Perspective", 언어연구, 29(1), pp.45~68 (2012), 언어정보연구소[DOI]
- 김종복, "English Cleft Constructions: Corpus Findings and Theoretical Implications", 생성문법연구, 17(2), pp.99~ (2007), 한국생성문법학회[DOI]
- Kiss, K. É, "Boundaries of Morphology and Syntax", pp.217~230 (1999), John Benjamins Publishing Co
- Kuncoro, A., "LSTMs Can Learn Syntax-Ssensitive Dependencies Well, But Modeling Structure Makes Them Better", Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, 1, pp.1426~1436 (2018)
- Lau, J. H., "Grammaticality, Acceptability, and Probability: A Probabilistic View of Linguistic Knowledge", Cognitive Science, 41(5), pp.1202~1241 (2017)
- 이규민, "DeepKLM - 통사 실험을 위한 전산 언어모델 라이브러리 -", 언어사실과 관점, 52, pp.265~306 (2021), 언어정보연구원[DOI]
- Levin, B., "English verb classes and alternations: A preliminary investigation" (1993), University of Chicago Press
- Levy, R, "Expectation-based Syntactic Comprehension", Cognition, 106(3), pp.1126~1177 (2008)
- Linzen, T, "What Can Linguistics and Deep Learning Contribute to Each Other? Response to Pater", Language, 95(1), pp.99~108 (2019)
- Linzen, T, "How Can We Accelerate Progress Towards Human-like Linguistic Generalization?", arXiv preprint arXiv:2005.00955, ~ (2020)
- Maier, G, "Corpus Interrogation and Grammatical Patterns", pp.173~205 (2014), John Benjamins Publishing Co
- Marvin, R., "Targeted Syntactic Evaluation of Language Models", arXiv preprint arXiv:1808.09031, ~ (2018)
- McCoy, T., "Right for the Wrong Reasons:Diagnosing Syntactic Heuristics in Natural Language Inference", arXiv preprint arXiv:1902.01007, ~ (2019)
- Kwonsik Park, "Deep learning can contrast the minimal pairs of syntactic data", 언어연구, 38(2), pp.395~424 (2021), 언어정보연구소[DOI]
- 박권식, "최소대립 문장쌍을 활용한 한국어 사전학습모델의 통사 연구 활용 가능성 검증", 언어와 정보, 25(3), pp.1~21 (2021), 한국언어정보학회
- Pater, J, "Generative Linguistics and Neural Networks at 60: Foundation, Friction, and Fusion", Language, 95(1), pp.41~74 (2019)
- Percus, O, "Prying Open the Cleft", North East Linguistics Society, 27(1), pp.1~15 (1997)
- Poliak, A., "Hypothesis Only Baselines in Natural Language Inference", preprint arXiv:1805.01042, ~ (2018)
- Quinn, H., "The Distribution of Pronoun Case Forms in English" (2005), John Benjamins Publishing Co
- Quirk, R., "A Comprehensive Grammar of the English Language" (1985), Longman
- Sobin, N, "Agreement, Default Rules, and Grammatical Viruses", Linguistic inquiry, 28(2), pp.318~343 (1997)
- 신운섭, "Is c-command Machine-learnable?", 언어학, 29(1), pp.183~204 (2021), 대한언어학회[DOI]
- Si, C., "What Does Bert Learn from Multiple-choice Reading Comprehension Datasets?", arXiv preprint arXiv:1910.12391, ~ (2019)
- Sornicola, R, "It-clefts and Wh-clefts: Two Awkward Sentence Types", Journal of Linguistics, 24(2), pp.343~379 (1988)
- Warstadt, A., "BLiMP: The Benchmark of Linguistic Minimal Pairs for English", Transactions of the Association for Computational Linguistics, 8, pp.377~392 (2020)
- Wilcox, E., "Structural Supervision Improves Learning of Non-Local Grammatical dependencies", arXiv preprint arXiv:1903.00943, ~ (2019)
- Yu, C., "Word Frequency Does Not Predict Grammatical Knowledge in Language Models", arXiv preprint arXiv:2010.13870, ~ (2020)