Affinity Prober를 이용한 언어 모델의 문장 수용성 판단 결정의 경계 요인 분석
Analysis on the Elements of the Decision Boundaries of Linguistic Acceptability in Language Model using Affinity Prober
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Recently, many studies endeavor to reveal the intrinsic nature of BERT, since transformer-based language models have achieved the state-of-the-art in many natural language understanding tasks. However, it is still hard to probe BERT’s universal linguistic properties since different probing methods lead to fluctuating results between tasks and models. Thus, this paper suggests Affinity Prober, which is a flexible task- and model-agnostic probing method to investigate transformer-based language model’s decision boundaries when processing linguistic phenomena. Affinity Prober is designed to investigate potential linguistic knowledge from the Self Attention Mechanism. Using Affinity Prober, the study examines whether a bert-base-cased model has any explainable decision boundaries on sentence acceptability in terms of a lexical category in English. The results in syntactic phenomena show that the Affinity Relationship between function words is reinforced in the upper layers, while the Affinity Relationship between content words plays a key role in semantic phenomena. The study concludes that Affinity Prober is helpful to analyze the model’s decision boundary with lexical categories on specific linguistic phenomena.
Abstract
Recently, many studies endeavor to reveal the intrinsic nature of BERT, since transformer-based language models have achieved the state-of-the-art in many natural language understanding tasks. However, it is still hard to probe BERT’s universal linguistic properties since different probing methods lead to fluctuating results between tasks and models. Thus, this paper suggests Affinity Prober, which is a flexible task- and model-agnostic probing method to investigate transformer-based language model’s decision boundaries when processing linguistic phenomena. Affinity Prober is designed to investigate potential linguistic knowledge from the Self Attention Mechanism. Using Affinity Prober, the study examines whether a bert-base-cased model has any explainable decision boundaries on sentence acceptability in terms of a lexical category in English. The results in syntactic phenomena show that the Affinity Relationship between function words is reinforced in the upper layers, while the Affinity Relationship between content words plays a key role in semantic phenomena. The study concludes that Affinity Prober is helpful to analyze the model’s decision boundary with lexical categories on specific linguistic phenomena.
참고문헌 (11)
- Devlin, Jacob, "BERT:Pre-training of deep bidirectional transformers for language understanding" (2019), North American Association for Computational Linguistics (NAACL)
- Htut, Phu Mon, "Do attention heads in BERT track syntactic dependencies?,” Computing Research Repository", arXiv:1911.12246 (2019)[DOI]
- Jawahar, Ganesh, "What Does BERT Learn about the Structure of Language?", Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.3651~3657 (2019), Association for Computational Linguistics
- Just, Marcel Adam, "Eye Movements in Reading: Perceptual and Language Processes" (1983), Academic Press
- Kim, Najoung, "Probing What Different NLP Tasks Teach Machines about Function Word Comprehension", Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), pp.235~249 (2019), Association for Computational Linguistics
- Kovaleva, Olga, "Revealing the Dark Secrets of BERT", Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp.4365~4374 (2019), Association for Computational Linguistics
- Rogers, Anna, "A Primer in BERTology : What We Know About How BERT Works", Transactions of the Association for Computational Linguistics, 8, pp.842~866 (2020)
- Warstadt, Alex, "Investigating BERT’s Knowledge of Language: Five Analysis Methods with NPIs", Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp.2877~2887 (2019), Association for Computational Linguistics
- Warstadt, Alex, "BLiMP: The Benchmark of Linguistic Minimal Pairs for English", Transactions of the Association for Computational Linguistics, 8, pp.377~392 (2020)
- Yu, Charles, "Word Frequency Does Not Predict Grammatical Knowledge in Language Models", Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.4040~4054 (2020), Association for Computational Linguistics
- Vaswani, Ashish, "Attention is allyou need", Advances in Neural Information Processing Systems, pp.6000~6010 (2017)