How are Korean Neural Language Models ‘surprised’ Layerwisely?
0
조회
0
다운로드
7
인용
0
참고문헌
키워드
초록
Since the introduction of BERT, recent works have shown success in detecting when a word is anomalous given sentence context. Since likelihood score is not an appropriate tool in identifying the exact property of linguistic anomaly, Li et al. (2021) recently adopt Gaussian models for density estimation at intermediate layers of pretrained language models. They find that different English pretrained language models employ separate mechanisms to recognize different types of linguistic anomaly. In keeping with Li et al.‘s methodology, we probe whether Korean counterparts such as KoBERT and KR-BERT are sensitive to different levels of linguistic anomaly, just as English-based language models are. To investigate the issue concerned, we construct an experiment with a suite of test data involving morphosyntactic, semantic, and commonsense anomaly in Korean and apply the two Korean-based models to test relevant sentences. We find that KoBERT and KR-BERT show relatively higher surprisal gaps throughout layers when the anomaly is morphosyntactic than when the anomaly is semantic. By contrast, commonsense anomaly does not exhibit any surprisal gap in any layer. We thus report that, like their English counterparts, KoBERT and KR-BERT use different mechanisms to track the different types of linguistic anomaly.
Abstract
Since the introduction of BERT, recent works have shown success in detecting when a word is anomalous given sentence context. Since likelihood score is not an appropriate tool in identifying the exact property of linguistic anomaly, Li et al. (2021) recently adopt Gaussian models for density estimation at intermediate layers of pretrained language models. They find that different English pretrained language models employ separate mechanisms to recognize different types of linguistic anomaly. In keeping with Li et al.‘s methodology, we probe whether Korean counterparts such as KoBERT and KR-BERT are sensitive to different levels of linguistic anomaly, just as English-based language models are. To investigate the issue concerned, we construct an experiment with a suite of test data involving morphosyntactic, semantic, and commonsense anomaly in Korean and apply the two Korean-based models to test relevant sentences. We find that KoBERT and KR-BERT show relatively higher surprisal gaps throughout layers when the anomaly is morphosyntactic than when the anomaly is semantic. By contrast, commonsense anomaly does not exhibit any surprisal gap in any layer. We thus report that, like their English counterparts, KoBERT and KR-BERT use different mechanisms to track the different types of linguistic anomaly.
참고문헌 (20)
- Choi, S. J., "KoBERT-wa KR-BERT-uy unnikchungpyel thongsa mich uymi cheli sengnung phyengka", Annual Conference on Human and Language Technology. Human and Language Technology, pp.340~345 (2021)
- Chomsky, N., "Syntactic Structures" (1957), Mouton
- Chow, W. Y., "A “bag-of-arguments” Mechanism for Initial Verb Predictions", Language, Cognition and Neuroscience, 31(5), pp.577~596 (2016)
- Devlin, J., "Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding", ~ (2018)
- Federmeier, K. D, "A Rose by Any Other Name:Long-term Memory Structure and Sentence Processing", Journal of Mmory and Language, 41(4), pp.469~495 (1999)
- Gulordava, K., "Colorless Green Recurrent Networks Dream Hierarchically", ~ (2018)
- Kelly, M. A., "Which Sentence Embeddings and Which Layers Encode Syntactic Structure?", Paper Delivered in Cogsci20 (2020)
- Kutas, M., "Handbook of Psycholinguistics", pp.659~724 (2006)
- Lee, S., "Kr-bert: A Small-scale Korean-specific Language Model", ~ (2020)
- Leech, G. N, "100 Million Words of English: the British National Corpus (BNC)", Language Research, 28(1), pp.1~13 (1992)
- Li, B., "How is BERT Surprised? Layerwise Detection of Linguistic Anomalies", ~ (2021)
- Liu, Y., "Roberta: A Robustly Optimized Bert Pretraining Approach", ~ (2019)
- Miaschi, A., "Linguistic Profiling of a Neural Language Model", ~ (2020)
- Osterhout, L, "Event-related Brain Potentials Elicited by Failure to Agree", Journal of Memory and Language, 34(6), pp.739~773 (1995)
- Osterhout, L, "On the Distinctiveness, Independence, and Time Course of the Brain Responses to Syntactic and Semantic Anomalies", Language and Cognitive Processes, 14(3), pp.283~317 (1999)
- Pylkkänen, L, "An MEG Study of Silent Meaning", Journal of Cognitive Neuroscience, 19(11), pp.1905~1921 (2007)
- Tenney, I., "BERT Rediscovers the Classical NLP Pipeline", ~ (2019)
- Warren, T., "Comprehending the Impossible: What Role do Selectional Restriction Violations Play?", Language, Cognition and Neuroscience, 30(8), pp.932~939 (2015)
- Warstadt, A., "BLiMP: The Benchmark of Linguistic Minimal Pairs for English", Transactions of the Association for Computational Linguistics, 8, pp.377~392 (2020)
- Yang, Z., "Xlnet: Generalized Autoregressive Pretraining for Language Understanding", ~ (2019)