다국어 범용 의존관계 주석체계를 준수하는 BERT 언어 모델 기반 자가-집중을 적용한 다중 작업 모델
A Multi-task Self-attention Model Using Pre-trained Language Models on Universal Dependency Annotations
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
본 논문에서는 UD Korean Kaist v2.3 코퍼스를 이용하여 범용 품사 태깅, 표제어추출 그리고 의존 구문분석을 동시에 예측할 수 있는 보편적 다중 작업 모델을 제안하였다. 제안 모델은 사전학습 언어모델인 다국어 BERT (Multilingual BERT)와 한국어 BERT (KR-BERT와 KoBERT)을 대상으로 추가학습 (fine-tuning)을 수행하여 BERT 모델의 자가-집중 (self-attention) 기법과 그래프 기반Biaffine attention 기법을 적용하여 제안 모델의 성능을 비교 분석하였다.
Abstract
In this paper, we propose a multi-task model that can simultaneously predict general-purpose tasks such as part-of-speech tagging, lemmatization, and dependency parsing using the UD Korean Kaist v2.3 corpus. The proposed model thus applies the self-attention technique of the BERT model and the graph-based Biaffine attention technique by fine-tuning the multilingual BERT and the two Korean-specific BERTs such as KR-BERT and KoBERT. The performances of the proposed model are compared and analyzed using the multilingual version of BERT and the two Korean-specific BERT language models.
참고문헌 (20)
- L. Joon-Ho, "Korean Dependency Guidelines for Dependency Parsing and Exo-Brain Language Analysis Corpus", proc. of the 27th Annual Conference on Human & Cognitive Language Technology, pp.224~239 (2015)
- "BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding"[URL]
- P. Cheoneum, "Korean Dependency Parsing with BERT", 2019 Proceedings of the Korean Information Science Society Conference, pp.530~532 (2019)
- 임준호, "Self-Attention 지배소 인식 모델을 이용한 어절 단위 한국어 의존 구문분석", 정보과학회논문지, 46(1), pp.22~30 (2019), 한국정보과학회[DOI]
- P. Seongsik, "Korean Dependency Parsing Using ELMo and Multi-head Attention", Proceedings of the 30th Annual Conference on Human and Cognitive Language Technology, pp.8~12 (2018)
- A. Hwijeen, "Korean Dependency Parsing Using Deep Bi-affine Network and Stack Pointer Network", Proceedings of the 30th Annual Conference on Human and Cognitive Language Technology, pp.689~691 (2018)
- L. Jinu, "Empirical Research on Segmentation Method for Korean Dependency parsing", Proceedings of the 33th Annual Conference on Human and Cognitive Language Technology, pp.427~432 (2021)
- 임준호, "사전학습 언어모델의 토큰 단위 문맥 표현을 이용한 한국어 의존 구문분석", 정보과학회논문지, 48(1), pp.27~34 (2021), 한국정보과학회[DOI]
- N. Seung-Hoon, "Deep Biaffine Attention for Korean Dependency Parsing", Proceedings of the Korean Information Science Society Conference, pp.584~586 (2017)
- J. Nivre, "Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection", Proceedings of the 12th Conference on Language Resources and Evaluation, pp.4034~4043 (2020)
- "Universal Dependencies"[URL]
- C. Jayeol, "Building Universal Dependency Treebanks in Korean", Proceedings of the 11th International Conference on Language Resources and Evaluation, pp.2194~2202 (2018)
- P. Hye-Jin, "Universal POS Tagset for Korean", Proceedings of the 30th Annual Conference on Human and Language Technology, pp.417~421 (2018)
- O. Taehwan, "Manual Revision of Penn Korean Universal Dependency Treebank", Proceedings of the 33th Annual Conference on Human and Cognitive Language Technology, pp.61~65 (2021)
- "KR-BERT"[URL]
- "KoBERT"[URL]
- M. Straka, "Evaluating Contextualized Embeddings on 54Languages in POS Tagging, Lemmatization and Dependency Parsing", ~
- D. Kondratyuk, "75 Languages, 1 Model: Parsing Universal Dependencies Universally", Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pp.2779~2795 (2019)
- T. Dozat, "Deep Biaffine Attention for Neural Dependency Parsing", ~