Exploring AI-Generated English Relative Clauses in Comparison to Human Production
0
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
Human behavioral studies have consistently indicated a preference for subject-extracted relative clauses (SRCs) over object-extracted relative clauses (ORCs) in sentence production and comprehension. Some studies have further shown that this preference can be influenced by the semantic properties of head nouns, particularly animacy. In this study, we use AI language models, specifically GPT-2 and ChatGPT 3.5, to simulate human sentence generation. Our primary goal is to evaluate the extent to which these language models replicate human behaviors in sentence production and identify any divergences. We tasked the models with completing sentence fragments structured as ‘the,’ followed by a head noun and 'that’ (The reporter that …). We varied the semantic property of head nouns such that they are all animate (the secretary that … ) in Study 1 and are either animate or inanimate (the musician/book that … ) in Study 2. Our findings reveal that in Study 1, both GPT models exhibited a robust SRC bias, replicating human-like behavior in relative clause production. However, in Study 2, we observed divergent behavior between the models when head nouns were inanimate, while consistency was maintained when head nouns were animate. Specifically, ChatGTP 3.5 generated more ORCs than SRCs in the presence of inanimate head nouns. These results, particularly those from ChatGPT 3.5, closely mirror human relative clause production patterns. Our study highlights the potential of language generative language models as efficient and versatile corpus simulators. Furthermore, our findings contribute to the evolving field of AI linguistics, shedding light on the capacity of AI generative systems to emulate human-like linguistic patterns in sentence production.
Abstract
Human behavioral studies have consistently indicated a preference for subject-extracted relative clauses (SRCs) over object-extracted relative clauses (ORCs) in sentence production and comprehension. Some studies have further shown that this preference can be influenced by the semantic properties of head nouns, particularly animacy. In this study, we use AI language models, specifically GPT-2 and ChatGPT 3.5, to simulate human sentence generation. Our primary goal is to evaluate the extent to which these language models replicate human behaviors in sentence production and identify any divergences. We tasked the models with completing sentence fragments structured as ‘the,’ followed by a head noun and 'that’ (The reporter that …). We varied the semantic property of head nouns such that they are all animate (the secretary that … ) in Study 1 and are either animate or inanimate (the musician/book that … ) in Study 2. Our findings reveal that in Study 1, both GPT models exhibited a robust SRC bias, replicating human-like behavior in relative clause production. However, in Study 2, we observed divergent behavior between the models when head nouns were inanimate, while consistency was maintained when head nouns were animate. Specifically, ChatGTP 3.5 generated more ORCs than SRCs in the presence of inanimate head nouns. These results, particularly those from ChatGPT 3.5, closely mirror human relative clause production patterns. Our study highlights the potential of language generative language models as efficient and versatile corpus simulators. Furthermore, our findings contribute to the evolving field of AI linguistics, shedding light on the capacity of AI generative systems to emulate human-like linguistic patterns in sentence production.
참고문헌 (58)
- Abdullah, M., "ChatGPT: fundamentals, applications and social impacts", Proceedings of the 2022 Ninth International Conference on Social Networks Analysis, pp.1~8 (2022)
- Gerry T.M Altmann, "Incremental interpretation at verbs: restricting the domain of subsequent reference", Cognition, 73(3), pp.247~264 (1999), Elsevier BV
- Arisoy, E., "Deep Neural Network Language Models", Proceedings of the NAACL-HLT 2012Workshop: Will We Ever Really Replace the N-gram Model? On the Future of Language Modeling for HLT, pp.20~28 (2012)
- Bernardy, J-P., "Using Deep Neural Networks to Learn Syntactic Agreement", Linguistic Issues in Language Technology, 15(2), pp.1~15 (2017)
- Bever, T. G., "The Cognitive Basis for Linguistic Structures", pp.278~362 (1970), John Wiley
- J.Kathryn Bock, "Syntactic persistence in language production", Cognitive Psychology, 18(3), pp.355~387 (1986), Elsevier BV[DOI]
- Cai, Z. G., "Does ChatGPT resemble humans in language use?", ArXiv., 2303.08014, pp.2023, ~
- Chomsky, N., "The false promise of ChatGPT", The New York Times, ~ (2023)
- Devlin, J., "BERT : Pre-training of deep bidirectional transformers for language understanding", Proceedings of the 2019 conference of the North American Chapter of the Association for Computational Linguistics : Human Language Technologies, 1, pp.4171~4186 (2019)
- Enguehard, É., "Exploring the Syntactic Abilities of RNNs with Multi-task Learning", Proceedings of the 21st Conference on Computational Natural Language Learning, pp.3~14 (2017)
- Ettinger, A., "Probing for semantic evidence of composition by means of simple classification tasks", Proceedings of the 1st Workshop on Evaluating Vector-space Representations for NLP, pp.134~139 (2016)
- Filippova, K., "Sentence compression by deletion with LSTMs", Proceedings of the 2015Conference on Empirical Methods in Natural Language Processing, pp.360~368 (2015)
- Ford, M., "A method for obtaining measures of local parsing complexity throughout sentences", Journal of Verbal Learning and Verbal Behavior, 22, pp.203~218 (1983)
- Fox, B. A., "A Discourse Explanation of the Grammar of Relative Clauses in English Conversation", Language, 66, pp.297~316 (1990)
- Frank, S. L., "The Interaction Between Structure and Meaning in Sentence Comprehension: Recurrent Neural Networks and Reading Times", Proceedings of the 2019 Cognitive Science Society, pp.337~343 (2019)
- Futrell, R., "RNNs as psycholinguistic subjects: Syntactic state and grammatical dependency", arXiv:1809.01329, ~ (2018)
- Futrell, R., "Do RNNs learn human-like abstract word order preferences", Proceedings of the 2019 Society for Computation in Linguistics (SCiL), pp.50~59 (2019)
- Gennari, S. P., "Semantic indeterminacy in object relative clauses", Journal of Memory and Language, 58(2), pp.161~187 (2008)
- Gibson, E., "Rational integration of noisy evidence and prior semantic expectations in sentence interpretation", Proceedings of the National Academy of Sciences, 110(20), pp.8051~8056 (2013)
- Gordon, P., "Memory interference during language processing", Journal of Experimental Psychology: Learning, Memory, and Cognition, 27(6), pp.1411~1423 (2001)
- Grodner, D., "Consequences of the Serial Nature of Linguistic Input for Sentenial Complexity", Cognitive Science, 29(2), pp.261~290 (2005)
- Gulordava, K., "Colorless green recurrent networks dream hierarchically", Proceedings of the 2018 conference of the north American chapter of the Association for Computational Linguistics: Human Language Technologies, 1, pp.1195~1205 (2018)
- Hale, J., "A probabilistic Earley parser as a psycholinguistic model", Proceedings of the second meeting of the North American Chapter of the Association for Computational Linguistics on Language Technologies (NAACL-HLT), pp.1~8 (2001)
- Heider, P. M., "The processing of it object relative clauses: Evidence against a fine-grained frequency account", Journal of Memory and Language, 75, pp.58~76 (2014)
- Howard, N., "BrainOS: A Novel Artificial Brain-Alike Automatic Machine Learning Framework", Frontiers in Computational Neuroscience, 14 (2020)
- Kako, E., "The semantics of syntactic frames", Language and Cognitive Processes, 21(5), pp.562~575 (2006)
- Kim, C. E., "Asymmetries in children's production of relative clauses : data from English and Korean", Journal of Child Language, 43(5), pp.1038~1071 (2016)
- King, M. R., "A Conversation on Artificial Intelligence, Chatbots, and Plagiarism in Higher Education", Cellular and Molecular Bioengineering, 16, pp.1~2 (2023)
- King, J., "Individual differences in syntactic processing:The role of working memory", Journal of Memory and Language, 30(5), pp.580~602 (1991)
- Levy, R., "Expectation-based syntactic comprehension", Cognition, 106, pp.1126~1177 (2008)
- Liebrenz, M., "Generating scholarly content with ChatGPT: ethical challenges for medical publishing", The Lancet Digital Health, 5(3), pp.e105~e106 (2023)
- Linzen, T., "Assessing the ability of LSTMs to learn syntax-sensitive dependencies", Transactions of the Association Computational Linguists, 4, pp.521~535 (2016)
- MacDonald, M. C., "How language production shapes language form and comprehension", Frontiers in Psychology, 4 (2013)
- MacDonald, M. C., "Reassessing working memory: Comment on Just and Carpenter (1992) and Waters and Caplan. 1996", Psychological Review, 109(1), pp.35~54 (2002)
- Mikolov, T., "Recurrent neural network based language model", pp.1045~1048 (2010), Interspeech
- Mitchell, D. C., "Exposure-based models of human parsing: Evidence for the use of coarse-grained (nonlexical)statistical records", Journal of Psycholinguist Research, 24, pp.469~488 (1995)
- [URL]
- Pickering, M. J., "The representation of verbs: Evidence from syntactic priming in language production", Journal of Memory and Language, 39(4), pp.633~651 (1998)
- Language models are unsupervised multitask learners, "Radford, A.", OpenAI tech, ~ (2019)
- Reali, F., "Processing of relative clauses is made easier by frequency of occurrence", Journal of Memory and Language, 57(1), pp.1~23 (2007)
- Roland, D., "The processing of pronominal relative clauses: Evidence from eye movements", Journal of Memory and Language, 119, pp.104244~ (2021)
- Roland, D., "Frequency of basic English grammatical structures: A corpus analysis", Journal of Memory and Language, 57(3), pp.348~379 (2007)
- Roland, D., "Discourse expectations and relative clause processing", Journal of Memory and Language, 66(3), pp.479~508 (2012)
- Rush, A. M., "A Neural Attention Model for Abstractive Sentence Summarization", Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp.379~389 (2015)
- Sallam, M., "ChatGPT utility in health care education, research, and practice: systematic review on the promising perspectives and valid concerns", Healthcare, 11(6)통권 : 887, pp.1~20 (2023)
- Schneider, E.T.R, "A GPT-2 language model for biomedical texts in Portuguese", Proceedings of the 2021 IEEE 34th International Symposium on ComputerBased Medical Systems (CBMS), pp.474~479 (2021)
- Schwenk, J., "High-resolution proteomics unravel architecture and molecular diversity of native AMPA receptor complexes", Neuron, 74(4), pp.621~633 (2012)
- Shin, U., "Investigating a neural language model’s replicability of psycholinguistic experiments: A case study of NPI licensing", Frontiers in Psychology, 14, pp.937656~ (2023)
- Shrivastava, A., "Enhancing aggression detection using GPT-2vbased data balancing technique", Proceedings of the 2021 5th International Conference on Intelligent Computing and Control Systems (ICICCS), pp.1345~1350 (2021)
- Traxler, M. J., "Processing subject and object relative clauses: Evidence from eye movements", Journal of Memory and Language, 47(1), pp.69~90 (2002)
- Thorp, H. H., "ChatGPT is fun, but not an author", Science, 379(6630), pp.313~313 (2023)
- Van Schijndel, M., "Modeling garden path effects without explicit hierarchical syntax", Proceedings of the 40th Annual Conference of the Cognitive Science Society, pp.2600~2605 (2018)
- Wang, F. Y., "What does chatGPT say: the DAO from algorithmic intelligence to linguistic intelligence", IEEE, 2023, pp.10(3), 575~579
- Warstadt, A., "Can neural networks acquire a structural bias from raw linguistic data?", arXiv:2007.06761, ~ (2020)
- Wilcox, E., "What do RNN language models learn about filler–gap dependencies?", Proceedings of the 2018EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP, pp.211~221 (2018)
- Wu, C., "Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models", arXiv:2303.04671, ~ (2023)
- 이은경, "An Experimental Investigation of Discourse Expectations in Neural Language Models", 영어학, 22, pp.1101~1115 (2022), 한국영어학회[DOI]
- 윤홍옥, "The Role of Frequency in the Processing of giving and receiving Events in Korean", 어학연구, 55(2), pp.253~279 (2019), 언어교육원[DOI]