Layer-wise Semantic Role Labeling with the KR-BERT Language Model
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
The purpose of this study is to assess the performance of semantic role labeling (SRL) predicted by the neural language models (NLMs, or Transformer-based pre-trained models) of Korean. First, the study built two models: the KR-BERT-BiLSTM-CRF model and the KR-BERT-Verb Position Feature (VPF)-BiLSTM-CRF model. The results from testing these two models show that the KR-BERT-VPF-BiLSTM-CRF model (67.3%) outperformed the KR-BERT-BiLSTM-CRF model (66.4%). In addition, this study examined which hidden layer improved the performance of NLMs during training. As expected, the NLM that was trained on the last hidden layer performed better than other alternative options such as the second-to-last-hidden layer and the concatenated last four layers. Thus, this study renders support to the general observation that an NLM should be trained on the last hidden layer to reach the highest performance. This study is meaningful since it is the first attempt to investigate which hidden layer is useful to train NLMs in SRL tasks of Korean.
Abstract
The purpose of this study is to assess the performance of semantic role labeling (SRL) predicted by the neural language models (NLMs, or Transformer-based pre-trained models) of Korean. First, the study built two models: the KR-BERT-BiLSTM-CRF model and the KR-BERT-Verb Position Feature (VPF)-BiLSTM-CRF model. The results from testing these two models show that the KR-BERT-VPF-BiLSTM-CRF model (67.3%) outperformed the KR-BERT-BiLSTM-CRF model (66.4%). In addition, this study examined which hidden layer improved the performance of NLMs during training. As expected, the NLM that was trained on the last hidden layer performed better than other alternative options such as the second-to-last-hidden layer and the concatenated last four layers. Thus, this study renders support to the general observation that an NLM should be trained on the last hidden layer to reach the highest performance. This study is meaningful since it is the first attempt to investigate which hidden layer is useful to train NLMs in SRL tasks of Korean.
참고문헌 (26)
- (2018)[URL]
- (2021)[URL]
- 배장성, "BERT를 이용한 한국어 의미역 결정", 정보과학회논문지, 47(11), pp.1021~1026 (2020), 한국정보과학회[DOI]
- Bazrafshan, M., "Semantic roles for string to tree machine translation", Proceedings of ACL, pp.419~423 (2013), Association for Computational Linguistics
- Berant, J., "Semantic parsing on freebase from question-answer pairs", Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (2013)
- Choi, S. J., "KRBERT impeyting chung-ey ttalu-n uymiyek kyelceng. (Layer-wise Semantic Role Labeling in KRBERT)", Annual Conference on Human and Language Technology, pp.617~621 (2021)
- Christensen, J., "Semantic role labeling for open information extraction", Proceedings of WS, pp.52~60 (2010), Associationfor Computational Linguistics
- Conia, S., "InVeRo-XL:Making cross-lingual semantic role labeling accessible with intelligible verbs and roles", Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp.319~328 (2021)
- Devlin, J., "BERT: Pre-training of deep bidirectional transformers for language understanding", arXiv preprint arXiv:1810.04805 (2018)[DOI]
- Marcheggiani, D., "A simple and accurate syntax-agnostic neural model for dependency-based semantic role labeling", Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pp.411~420 (2017)
- 이성희, "원거리 감독과 능동 배깅을 이용한 개체명 인식", 정보과학회논문지, 43(2), pp.269~274 (2016), 한국정보과학회
- Lee, S., "Kr-bert: A small-scale Korean-specific language model", arXiv preprint arXiv:2008.03979, pp.03979~ (2016)
- Li, Z., "A unified syntax-aware framework for semantic role labeling", Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.2401~2411 (2018)
- Liu, N. F., "Linguistic knowledge and transferability of contextual representations", arXiv preprint arXiv: 1903.08855, pp.08855~ (2019)
- Liu, D., "Semantic role features for machine translation", Proceedings of COLING, pp.716~724 (2010), Coling 2010 Organizing Committee
- Lin, Y., "Neural relation extraction with multi-lingual attention", Proceedings of ACL, pp.34~43 (2017), Association for Computational Linguistics
- Park, S., "Grapheme-level awareness in word embeddings for morphologically rich languages", Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) (2018)
- Peters, M. E., "Dissecting contextual word embeddings: Architecture and representation", arXiv preprint arXiv:1808.08949, pp.08949~ (2018)
- Roth, M., "Neural semantic role labeling with dependency path embeddings", Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 1, pp.1192~1202 (2016)
- Shi, P., "Simple bert models for relation extraction and semantic role labeling", arXiv preprint arXiv:1904.05255, pp.05255~ (2019)
- Shi, X., "Does string-based neural MT learn source syntax?", Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp.1526~1534 (2016)
- Tenney, I., "BERT rediscovers the classical NLP pipeline", arXiv preprint arXiv: 1905.05950 (2019)
- Yih, W., "The value of semantic parse labeling for knowledge base question answering", Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 2 (2016)
- Zhang, Y., "Graph convolution over pruned dependency trees improves relation extraction", Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.2205~2215 (2018)
- Zhang, Z., "Comparing span extraction methods for semantic role labeling", Proceedings of the 5th Workshop on Structured Prediction for NLP (SPNLP 2021), pp.67~77 (2021)
- Zhang, Y., "Semantic role labeling as dependency parsing: Exploring latent tree structures inside arguments", arXiv preprint arXiv:2110.06865 (2021)