딥러닝 언어모형의 평가와 언어학
Evaluation of the Deep Learning-based Language Models and Linguistics
1
조회
0
다운로드
3
인용
0
참고문헌
키워드
초록
This article addresses how the deep learning-based language models can be evaluated with respect to linguistic knowledge. Building upon the overlook, this article discusses how linguistics can make a substantial contribution to the development of the artificial intelligence systems. As many transformer-based models have been competitively implemented for the last few years, it is required to evaluate the multiple models in a common and reliable way. For this purpose, a wide range of linguistic evaluation metrics have been designed and constructed. The evaluation datasets involve the concepts used in theoretical linguistics, such as syntax, semantics, and pragmatics. The evaluation process follows the guideline used in psycholinguistic experiments. As such, the linguistic knowledge enhances interpretability of the deep leaning-based natural language processing techniques. It is contended that linguistics will play a pivotal role in evaluating and improving the language models in further research.
Abstract
This article addresses how the deep learning-based language models can be evaluated with respect to linguistic knowledge. Building upon the overlook, this article discusses how linguistics can make a substantial contribution to the development of the artificial intelligence systems. As many transformer-based models have been competitively implemented for the last few years, it is required to evaluate the multiple models in a common and reliable way. For this purpose, a wide range of linguistic evaluation metrics have been designed and constructed. The evaluation datasets involve the concepts used in theoretical linguistics, such as syntax, semantics, and pragmatics. The evaluation process follows the guideline used in psycholinguistic experiments. As such, the linguistic knowledge enhances interpretability of the deep leaning-based natural language processing techniques. It is contended that linguistics will play a pivotal role in evaluating and improving the language models in further research.
참고문헌 (31)
- Bender, E. M., "Integrating ethics into the NLP curriculum", Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020)
- Cer, D., "Semeval2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation", arXiv preprint arXiv:1708.00055 (2017)
- De Marneffe, M-C., "The commitmentbank:Investigating projection in naturally occurring discourse", Proceedings of Sinn und Bedeutung, 23(2) (2019)
- Devlin, J., "Bert: Pre-training of deep bidirectional transformers for language understanding", arXiv preprint arXiv:1810.04805 (2018)
- Dunbar, E., "Generative grammar, neural networks, and the implementational mapping problem: Response to Pater", Language, 95(1), pp.e87~e98 (2019)
- Gururangan, S., "Annotation artifacts in natural language inference data", arXiv preprint arXiv:1803.02324 (2018)
- Huang, L., "COSMOS QA:Machine reading comprehension with contextual commonsense reasoning", arXiv preprint arXiv:1909.00277 (2019)
- Jin, D., "Is BERT really robust? a strong baseline for natural language attack on text classification and entailment", Proceedings of the AAAI conference on artificial intelligence (2020)
- Koehn, P., "Statistical Machine Translation" (2009), Cambridge University Press
- Le, H., "FlauBERT: Unsupervised language model pre-training for french", arXiv preprint arXiv:1912.05372 (2019)
- Lee, J., "BioBERT: a pre-trained biomedical language representation model for biomedical text mining", Bioinformatics, 36(4), pp.1234~1240 (2020)
- Linzen, T., "What can linguistics and deep learning contribute to each other? Response to Pater", Language, 95(1), pp.e99~e108 (2019)
- McCoy, R. T., "Revisiting the poverty of the stimulus: Hierarchical generalization without a hierarchical bias in recurrent neural networks", Proceedings of the 40th annual conference of the Cognitive Science Society (2018)
- Nie, Y., "Adversarial NLI: A new benchmark for natural language understanding", arXiv preprint arXiv:1910.14599 (2019)
- Pakhomov, S., "Semantic similarity and relatedness between clinical terms: an experimental study", AMIA annual symposium proceedings (2010)
- Park, S., "KLUE: Korean Language Understanding Evaluation", arXiv preprint arXiv:2105.09680 (2021)
- Parrish, A., "Does Putting a Linguist in the Loop Improve NLU Data Collection?", Findings of the Association for Computational Linguistics: EMNLP 2021 (2021)
- Pater, J., "Generative linguistics and neural networks at 60: Foundation, friction, and fusion", Language, 95(1), pp.e41~e74 (2019)
- Pearl, L. S., "Fusion is great, and interpretable fusion could be exciting for theory generation: Response to Pater", Language, 95(1), pp.e109~e114 (2019)
- Pruksachatkun, Y., "jiant: A software toolkit for research on general-purpose text understanding models", arXiv preprint arXiv:2003.02249 (2020)
- Rawski, J., "No free lunch in linguistics or machine learning:Response to pater", Language, 95(1), pp.e125~e135 (2019)
- Rudinger, R., "Gender bias in coreference resolution", Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2018)
- Wang, A., "Superglue: A stickier benchmark for generalpurpose language understanding systems analysis", arXiv preprint arXiv:1905.00537 (2019)
- Wang, A., "GLUE: A multi-task benchmark and analysis platform for natural language understanding", arXiv preprint arXiv:1804.07461 (2018)
- Wang, Y., "MedSTS: a resource for clinical semantic textual similarity", Language Resources and Evaluation, 54(1), pp.57~72 (2020)
- Warstadt, A., "Neural network acceptability judgments", Transactions of the Association for Computational Linguistics, 7, pp.625~641 (2019)
- Williams, A., "A broad-coverage challenge corpus for sentence understanding through inference", arXiv preprint arXiv:1704.05426 (2017)
- Xu, L., "CLUE: A chinese language understanding evaluation benchmark", Proceedings of the 28th International Conference on Computational Linguistics, pp.4762~4772 (2020)
- Yum, Y., "A Word Pair Dataset for Semantic Similarity and Relatedness in Korean Medical Vocabulary: Reference Development and Validation", JMIR Medical Informatics, 9(6), pp.e29667~ (2021)
- Zellers, R., "SWAG: A large-scale adversarial dataset for grounded commonsense inference", arXiv preprint arXiv:1808.05326 (2018)
- Zhang, Y., "BioWordVec, improving biomedical word embeddings with subword information and MeSH", Scientific data, 6, pp.1~9 (2019)