언어 학습자로서의 신경망 언어모델: 영어 주어-동사 일치를 중심으로
A Neural Language Model as a Language Learner: Focusing on Subject-Verb Agreement
1
조회
0
다운로드
2
인용
0
참고문헌
키워드
초록
This paper assesses the language-processing ability of the neural language model (L2-GPT-2) trained on data sets of English textbooks published in Korea using the Generative Pre-trained Transformer (GPT)-2. Assuming that the language model (LM) is also an (artificial) language learner, we test it focusing on subject-verb agreement in English. It is a well-established fact that L1 speakers exhibit facilitatory interference effects in ungrammatical sentences with a plural subject and a singular form of verb. Unlike human native speakers, L2-GPT-2 as well as L1-GPT-2 display such effects in ungrammatical sentences either with a plural subject and a singular form of verb, or with a singular subject and a plural form of verb. Though there is a significant difference between human speakers and neural LMs in processing subject-verb agreement, the two LMs’ sensitivity to interference by a distractor NP points to the fact that they can attain a remarkably human-like linguistic generalization on subject-verb agreement.
Abstract
This paper assesses the language-processing ability of the neural language model (L2-GPT-2) trained on data sets of English textbooks published in Korea using the Generative Pre-trained Transformer (GPT)-2. Assuming that the language model (LM) is also an (artificial) language learner, we test it focusing on subject-verb agreement in English. It is a well-established fact that L1 speakers exhibit facilitatory interference effects in ungrammatical sentences with a plural subject and a singular form of verb. Unlike human native speakers, L2-GPT-2 as well as L1-GPT-2 display such effects in ungrammatical sentences either with a plural subject and a singular form of verb, or with a singular subject and a plural form of verb. Though there is a significant difference between human speakers and neural LMs in processing subject-verb agreement, the two LMs’ sensitivity to interference by a distractor NP points to the fact that they can attain a remarkably human-like linguistic generalization on subject-verb agreement.
참고문헌 (44)
- Bernardy, Jean-Philippe, "Using deep neural networks to learn syntactic agreement", Linguistic Issues in Language Technology, 15, pp.1~15 (2017)
- Bock, Kathryn, "Broken agreement", Cognitive Psychology, 23(1), pp.45~93 (1991)
- Clifton, C., "Feature manipulationins entence comprehension", Riv. Linguist, 11, pp.11~39 (1999)
- Dillon, Brian, "Contrasting intrusion profiles for agreement and anaphora : Experimental and modeling evidence", Journal of Memory and Language, 69(2), pp.85~103 (2013)
- Drenhaus, H., "Linguistic Evidence: Empirical, Theoretical, and Computational Perspectives", pp.145~165 (2005), deGruyter
- Eberhard, K. M., "Making syntax of sense : number agreement in sentence production", Psychological Review, 112, pp.531~539 (2005)
- Engelmann, Felix, "The effect of prominence and cue association in retrieval processes: A computational account", Cognitive Science Cognitive Science, 43, pp.e12800~ (2019)
- The determinants of retrieval interference in dependency resolution: Review and computational modeling, "Engelmann, Felix", Computer science, ~ (2015)
- Enguehard, Emile, "Exploring the syntactic abilities of RNNs with multi-task learning", arXiv preprint arXiv:1706.03542, ~ (2017)
- Frank, Stefan L., "The ERP response to the amount of information conveyed by words in sentences", Brain and Language, 140, pp.1~11 (2015)
- Futrell, Richard, "Rnns as psycholinguistic subjects: Syntactic state and grammatical dependency", arXiv preprint arXiv:1809.01329, ~ (2018)
- Futrell, Richard, "Neural language models as psycholinguistic subjects: Representations of syntactic state", Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, pp.32~42 (2019)
- Gulordava, Kristina, "Colorless green recurrent networks dream hierarchically" (2018), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
- Hale, John, "A probabilistic earley parser as a psycholinguistic model", Second Meeting of the North American Chapter of the Association for Computational Linguistics (2001)
- Jager, Lena A., "Similarity-based interference in sentence comprehension : Literature review and Bayesian meta-analysis", Journal of Memory and Language, 94, pp.316~339 (2017)
- Klafka, Josef, "Spying on your neighbors: Fine-grained probing of contextual embeddings for information about surrounding words", arXiv preprint arXiv:2005.01810, ~ (2020)
- Lago, Sol, "Agreement attraction in spanish comprehension", Journal of Memory and Language, 82, pp.133~149 (2015)
- Lau, E. F., "Syntactic prediction and lexical surface frequency effects in sentence processing", University of Maryland Working Papers in Linguistics, 16, pp.163~200 (2007)
- Lehtonen, M., "Recognition of inflected words in a morphologically limited language : Frequency effects in monolinguals and bilinguals", Journal of Psycholinguistic Research, 35(2), pp.121~146 (2006)
- Levy, Roger, "Expectation-based syntactic comprehension", Cognition, 106(3), pp.1126~1177 (2008)
- Lewis, Richard L., "Computational principles of working memory in sentence comprehension", Trends in cognitive sciences, 10(10), pp.447~454 (2006)
- Lewis, Richard L., "An activation-based model of sentence processing as skilled memory retrieval", Cognitive science, 29(3), pp.375~419 (2005)
- Linzen, Tal, "Syntactic structure from deep learning", Annual Review of Linguistics (2021)
- Linzen, Tal, "Assessing the ability of lstms to learn syntaxsensitive dependencies", Transactions of the Association for Computational Linguistics, 4, pp.521~535 (2016)
- Logačev, P., "A multiple-channel model of task-dependent ambiguity resolution in sentence comprehension", Cognitive Science, 40(2), pp.266~298 (2016)
- McElree, B., "Memory structures that subserve sentence comprehension", Journal of Memory and Language, 48, pp.67~91 (2003)
- Misra, K., "Exploring BERT's Sensitivity to Lexical Cues using Tests from Semantic Priming", arXiv preprint arXiv:2010.03010, ~ (2020)
- New, B., "The processing of singular and plural nouns in french and english", Journal of Memory and Language, 51(4), pp.568~585 (2004)
- Pearlmutter, Neal J, "Agreement processes in sentence comprehension", Journal of Memory and language, 41(3), pp.427~456 (1999)
- Radford, Alec, "Language models are unsupervised multitask learners", OpenAI blog, 1(8), pp.9~ (2019)
- Rye, Soo Hyun, "Accounting for Agreement Phenomena in Sentence Comprehension with Transformer Language Models: Effects of Similarity-based Interference on Surprisal and Attention", arXiv-CS-Computation and Language (IF), ~ (2021)
- Smith, Nathaniel J., "The effect of word predictability on reading time is logarithmic", Cognition, 128(3), pp.302~319 (2013)
- Staub, Adrian, "On the interpretation of the number attraction effect : Response time evidence", Journal of memory and language, 60, pp.308~327 (2009)
- Staub, Adrian, "Response time distributional evidence for distinct varieties of number attraction", Cognition, 114, pp.447~454 (2010)
- Van Dyke, J., "Interference effects from grammatically unavailable constituents during sentence processing", Journal of Experimental Psychology. Learning, Memory, and Cognition, 33(2), pp.407~430 (2007)
- Van Dyke, Julie A., "Distinguishing effects of structure and decay on attachment and repair : A cue-based parsing account of recovery from misanalyzed ambiguities", Journal of Memory and Language, 49(3), pp.285~316 (2003)
- Van Dyke, Julie A., "Cue-dependent interference in comprehension", Journal of Memory and Language, 65(3), pp.247~263 (2011)
- Van Schijndel, Marten, "Modeling garden path effects without explicit hierarchical syntax", CogSci (2018)
- Vasishth, S., "Processing polarity : How the ungrammatical intrudes on the grammatical", Cognitive Science, 32, pp.685~712 (2008)
- Vaswani, Ashish, "Attention is all you need", Advances in neural information processing systems, pp.5998~6008 (2017)
- Wagers, Matthew W, "Agreement attraction in comprehension : Representations and processes", Journal of Memory and Language, 61(2), pp.206~237 (2009)
- Wilcox, Ethan, "On the predictive power of neural language models for human real-time comprehension behavior", arXiv preprint arXiv:2006.01912, ~ (2020)
- Xiang, M., "Illusory licensing effects across dependency types : ERP evidence", Brain Language, 108, pp.40~55 (2009)
- Yu, L., "Assessing Phrasal Representation and Composition in Transformers" (2020), W. H. Freeman and Company