KorPatELECTRA : 자연어처리 분야에서의 성능 향상을 위한 한국어 특허 문헌 사전학습 언어모델(KorPatELECTRA)
KorPatELECTRA : A Pre-trained Language Model for Korean Patent Literature to improve performance in the field of natural language processing(Korean Patent ELECTRA)
1
조회
0
다운로드
4
인용
0
참고문헌
키워드
초록
특허 분야에서 자연어처리(Natural Language Processing) 태스크는 특허문헌의 언어적 특이성으로문제 해결의 난이도가 높은 과제임에 따라 한국 특허문헌에 최적화된 언어모델의 연구가 시급한실정이다. 최근 자연어처리 분야에서는 특정 도메인에 특화되게 사전 학습(Pre-trained)한 언어모델을 구축하여 관련 분야의 다양한 태스크에서 성능을 향상시키려는 시도가 지속적으로 이루어지고있다. 그 중, ELECTRA는 Google이 BERT 이후에 RTD(Replaced Token Detection)라는 새로운 방식을 제안하며 학습 효율성을 높인 사전학습 언어모델이다. 본 연구에서는 대량의 한국 특허문헌데이터를 사전 학습한 KorPatELECTRA를 제안한다. 또한, 특허 문헌의 특성에 맞게 학습 코퍼스를 정제하고 특허 사용자 사전 및 전용 토크나이저를 적용하여 최적화된 사전 학습을 진행하였다. KorPatELECTRA의 성능 확인을 위해 실제 특허데이터를 활용한 NER(Named Entity Recognition), MRC(Machine Reading Comprehension), 특허문서 분류 태스크를 실험하였고 비교 대상인 범용 모델에 비해 3가지 태스크 모두에서 가장 우수한 성능을 확인하였다.
Abstract
In the field of patents, as NLP(Natural Language Processing) is a challenging task due to the linguistic specificity of patent literature, there is an urgent need to research a language model optimized for Korean patent literature. Recently, in the field of NLP, there have been continuous attempts to establish a pre-trained language model for specific domains to improve performance in various tasks of related fields. Among them, ELECTRA is a pre-trained language model by Google using a new method called RTD(Replaced Token Detection), after BERT, for increasing training efficiency. The purpose of this paper is to propose KorPatELECTRA pre-trained on a large amount of Korean patent literature data. In addition, optimal pre-training was conducted by preprocessing the training corpus according to the characteristics of the patent literature and applying patent vocabulary and tokenizer. In order to confirm the performance, KorPatELECTRA was tested for NER(Named Entity Recognition), MRC(Machine Reading Comprehension), and patent classification tasks using actual patent data, and the most excellent performance was verified in all the three tasks compared to comparative general-purpose language models.
참고문헌 (21)
- 정수정, "독일어와 한국어 특허문서에 쓰이는 고정적 언어 표현에 대한 연구", 독일어문화권연구, (23), pp.359~381 (2014), 독일어문화권연구소
- DEVLIN, Jacob, "Bert: Pre-training of deep bidirectional transformers for language understanding", ~ (2018)
- Yang, Zhilin, "Xlnet: Generalized autoregressive pretraining for language understanding", Advances in neural information processing systems, 32 (2019)
- RADFORD, Alec, "Improving language understanding by generative pre-training", ~ (2018)
- LAN, Zhenzhong, "A Lite BERT for Self-supervised Learning of Language Representations", ~ (2019)
- LIU, Yinhan, "Roberta: A robustly optimized bert pretraining approach", ~ (2019)
- LIM, J. H., "Recent R&D Trends for Pretrained Language Model", Electronics and Telecommunications Trends, 35(3), pp.9~19 (2020)
- CLARK, Kevin, "Electra: Pre-training text encoders as discriminators rather than generators", ~ (2020)
- WANG, Alex, "GLUE: A multi-task benchmark and analysis platform for natural language understanding", ~ (2018)
- LEE, Jinhyuk, "BioBERT: a pre-trained biomedical language representation model for biomedical text mining", Bioinformatics, 36(4), pp.1234~1240 (2020)
- BELTAGY, Iz, "Scibert: A pretrained language model for scientific text", ~ (2019)
- LEWIS, Patrick, "Pretrained Language Models for Biomedical and Clinical Tasks: Understanding and Extending the State-of-the-Art", Proceedings of the 3rd Clinical Natural Language Processing Workshop, pp.146~157 (2020)
- RAJ KANAKARAJAN, Kamal, "BioELECTRA: Pretrained Biomedical text Encoder using Discriminators", Proceedings of the 20th Workshop on Biomedical Language Processing, pp.143~154 (2021)
- VASWANI, Ashish, "Advances in neural information processing systems", pp.5998~6008 (2017)
- 민재옥, "BERT를 이용한 한국어 특허상담 기계독해", 정보처리학회 논문지, 9(4), pp.145~152 (2020), 한국정보처리학회[DOI]
- Park, Joo-Yeon, "Improving Recognition of Patent’s Claims with Deep Neural Networks", Collection of papers from Korea Information Processing Society, 27(1), pp.500~503 (2020)
- LEE, Jieh-Sheng, "Patent classification by fine-tuning BERT language model", World Patent Information, 61, pp.101965~ (2020)
- RUST, Phillip, "How good is your tokenizer? on the monolingual performance of multilingual language models", ~ (2020)
- SENNRICH, Rico, "Neural machine translation of rare words with subword units", ~ (2015)
- PARK, Sungjoon, "KLUE: Korean Language Understanding Evaluation", ~ (2021)
- PARK, Jinwoo, "Patent Tokenizer: a research on the optimization of tokenize for the Patent sentence using the Morphemes and SentencePiece", Annual Conference on Human and Language Technology. Human and Language Technology, pp.441~445 (2020)