적대적 사례에 기반한 언어 모형의 한국어 격 교체 이해 능력 평가
Adversarial Example-Based Evaluation of How Language Models Understand Korean Case Alternation
1
조회
0
다운로드
3
인용
0
참고문헌
키워드
초록
In the field of deep learning-based language understanding, adversarial examples refer to deliberately constructed examples of data, slightly different from original examples. The contrasts between the original and adversarial examples are less perceivable to human readers, but the disruption has a notorious effect on the performance of machines. Thus, adversarial examples facilitate assessing whether and how a specific deep learning architecture (e.g., a language model) robustly works. Out of the multiple layers of linguistic structures, this study lays focus on a morphosyntactic phenomenon in Korean, namely, case alternation. We created a set of adversarial examples regarding case alternation, and then tested the morpho-syntactic ability of neural language models. We extracted the instances of case alternation from the Sejong Electronic Dictionary, and made use of mBERT and KR-BERT as the language models. The results (measured by means of surprisal) indicate that the language models are unexpectedly good at discerning case alternation in Korean. In addition, it turns out that the Korean-specific language model performs better than the multilingual model. These imply that an in-depth knowledge of linguistics is essential for creating adversarial examples in Korean.
Abstract
In the field of deep learning-based language understanding, adversarial examples refer to deliberately constructed examples of data, slightly different from original examples. The contrasts between the original and adversarial examples are less perceivable to human readers, but the disruption has a notorious effect on the performance of machines. Thus, adversarial examples facilitate assessing whether and how a specific deep learning architecture (e.g., a language model) robustly works. Out of the multiple layers of linguistic structures, this study lays focus on a morphosyntactic phenomenon in Korean, namely, case alternation. We created a set of adversarial examples regarding case alternation, and then tested the morpho-syntactic ability of neural language models. We extracted the instances of case alternation from the Sejong Electronic Dictionary, and made use of mBERT and KR-BERT as the language models. The results (measured by means of surprisal) indicate that the language models are unexpectedly good at discerning case alternation in Korean. In addition, it turns out that the Korean-specific language model performs better than the multilingual model. These imply that an in-depth knowledge of linguistics is essential for creating adversarial examples in Korean.
참고문헌 (39)
- 김미령, "격교체 양상에 따른 동사 분류에 대한 연구 -‘-에", 를’ 교체 동사의 특성과 분류를 중심으로-, 2004, pp.(25), 161~190[DOI]
- 박권식, "최소대립 문장쌍을 활용한 한국어 사전학습모델의 통사 연구 활용 가능성 검증", 언어와 정보, 25(3), pp.1~21 (2021), 한국언어정보학회
- 송창선, "격조사 교체 현상을 통해 본 국어의 격 기능", 국어교육연구, (71), pp.21~38 (2019), 국어교육학회(since1969)[DOI]
- 우형식, "국어의 타동사 구문 연구" (1996), 도서출판 박이정
- 이규민, "DeepKLM - 통사 실험을 위한 전산 언어모델 라이브러리 -", 언어사실과 관점, 52, pp.265~306 (2021), 언어정보연구원[DOI]
- 이종근, "한국어 동사와 대격에 관한 연구", 언어학, 14(1), pp.223~242 (2006), 대한언어학회
- 이홍식, "조사 ‘을’의 의미에 대하여", 한국어 의미학, 15, pp.303~327 (2004), 한국어의미학회
- "KorBERT(Korean Bidirectional Encoder Representations from Transformers)" (2019), 한국전자통신연구원[URL]
- 홍재성, "세종 전자사전 : 전산어휘부로서의 특성과 의의", 한국정보과학회언어공학 연구회 학술발표 논문집, pp.323~331 (2007)
- Bender, E. M., "Linguistically naïve != language independent: Why NLP needs linguistic typology", Paper presented at the EACL 2009 Workshop on the Interaction between Linguistics and Computational Linguistics: Virtuous, Vicious or Vacuous?, pp.26~32 (2009)
- Da Costa, J. K., "Assessing the ability of Transformer-based Neural Models to represent structurally unbounded dependencies", Paper presented at the Society for Computation in Linguistics, 3(1), pp.189~198 (2020)
- Devlin, J., "BERT: Pre-training of deep bidirectional transformers for language understanding", Paper presented at the 2019Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.4171~4186 (2019)
- Ebrahimi, J., "On adversarial examples for character- level neural machine translation", Paper presented at the 27th International Conference on Computational Linguistics, pp.653~663 (2018)
- Fukuda, S., "The syntax of variable behavior verbs : Experimental evidence from the accusative–oblique alternations in Japanese", Journal of Linguistics, 56(2), pp.269~314 (2020)
- Garg, S., "BAE: BERT-based adversarial examples for text lassification", Paper presented at the 2020 Conference on Empirical Methods in Natural Language Processing, pp.6174~6181 (2020)
- Goldberg, Y., "Assessing BERT's syntactic abilities", arXiv preprint arXiv:1901.05287, ~ (2019)
- Goodfellow, I. J., "Explaining and harnessing adversarial examples", Paper presented at the ICLR 2015 (2015)
- Hale, J., "A probabilistic Earley parser as a psycholinguistic model", Paper presented at the Second meeting of the north American chapter of the association for computational linguistics (2001)
- Hu, J., "A systematic assessment of syntactic generalization in neural language models", Paper presented at the Association for Computational Linguistics, pp.1725~1744 (2020)
- Jeretic, P., "Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition", Paper presented at the 58th Annual Meeting of the Association for Computational Linguistics, pp.8690~8705 (2020)
- Jiang, N., "He Thinks He Knows Better than the Doctors:BERT for Event Factuality Fails on Pragmatics", Transactions of the Association for Computational Linguistics, 9, pp.1081~1097 (2021)
- Jin, D., "Is BERT really robust? a strong baseline for natural language attack on text classification and entailment", Paper presented at the AAAI conference on artificial intelligence, 34(5), pp.8018~8025 (2020)
- Lee, S., "KR-BERT: A small-scale korean-specific language model", arXiv preprint arXiv:2008.03979, ~ (2020)
- Levy, R., "Expectation-based syntactic comprehension", Cognition, 106(3), pp.1126~1177 (2008)
- Marvin, R., "Targeted syntactic evaluation of language models", Paper presented at the 2018 Conference on Empirical Methods in Natural Language Processing, pp.1192~1202 (2018)
- Meister, C., "Revisiting the uniform information density hypothesis", arXiv preprint arXiv:2109.11635, ~ (2021)
- Nie, Y., "Adversarial NLI: A new benchmark for natural language understanding", Paper presented at the 58th Annual Meeting of the Association for Computational Linguistics, pp.4885~4901 (2019)
- Kwonsik Park, "Deep learning can contrast the minimal pairs of syntactic data", 언어연구, 38(2), pp.395~424 (2021), 언어정보연구소[DOI]
- 박상희, "Perception-production asymmetry for Korean double accusative ditransitives", 언어연구, 38(1), pp.27~52 (2021), 언어정보연구소[DOI]
- Pires, T., "How multilingual is multilingual BERT?", Paper presented at the 57th Annual Meeting of the Association for Computational Linguistics, pp.4996~5001 (2019)
- Sinha, K., "Masked language modeling and the distributional hypothesis: Order word matters pre-training for little", Paper presented at the 2021 Conference on Empirical Methods in Natural Language Processing, pp.2888~2913 (2021)
- Szegedy, C., "Intriguing properties of neural networks", Paper presented at International Conference on Learning Representations (ICLR) (2014)
- Taylor, W. L., ""Cloze procedure" : A new tool for measuring readability", Journalism quarterly, 30(4), pp.415~433 (1953)
- Wei, J., "Frequency Effects on Syntactic Rule Learning in Transformers", Paper presented at the 2021 Conference on Empirical Methods in Natural Language Processing, pp.932~948 (2021)
- Wilcox, E., "Hierarchical representation in neural language models: Suppression and recovery of expectations", arXiv preprint arXiv:1906.04068, ~ (2019)
- Wu, Y., "Google's neural machine translation system: Bridging the gap between human and machine translation", arXiv preprint arXiv:1609.08144, ~ (2016)
- Yanaka, H., "Assessing the Generalization Capacity of Pre-trained Language Models through Japanese Adversarial Natural Language Inference", Paper presented at the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp.337~349 (2021)
- Yu, C., "Word frequency does not predict grammatical knowledge in language models", Paper presented at the 2020Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.4040~4054 (2020)
- Zellers, R., "SWAG: A large-scale adversarial dataset for grounded commonsense inference", Paper presented at the 2018Conference on Empirical Methods in Natural Language Processing, pp.93~104 (2018)