딥러닝 기반 사전학습 언어모델에 대한 이해와 현황
A Survey on Deep Learning-based Pre-Trained Language Models
1
조회
0
다운로드
4
인용
0
참고문헌
키워드
초록
사전학습 언어모델은 자연어 처리 작업에서 가장 중요하고 많이 활용되는 도구로, 대량의 말뭉치를 대상으로 사전학습이 되어있어 적은 수의 데이터를 이용한 미세조정학습으로도 높은 성능을 기대할 수 있으며, 사전학습된 토크나이저과 딥러닝 모형 등 구현에 필요한 요소들이 함께 배포되기 때문에 자연어 처리 작업에 소요되는 비용과 기간을 크게 단축시켰다. 트랜스포머 변형 모형은 이와 같은 장점을 제공하는 사전학습 언어모델 중에서 최근 가장 많이 사용되고 있는 모형으로, 번역을 비롯하여 문서 요약, 챗봇과 같은 질의 응답, 자연스러운 문장의 생성 및 문서의 분류 등 다양한 자연어 처리 작업에 활용되고 있으며 컴퓨터 비전 분야와 오디오 관련 분야 등 다른 분야에서도 활발하게 활용되고 있다. 본 논문은 연구자들이 보다 쉽게 사전학습 언어모델에 대해 이해하고 자연어 처리 작업에 활용할 수 있도록 하기 위해, 언어모델과 사전학습 언어모델의 정의로부터 시작하여 사전학습 언어모델의 발전과정과 다양한 트랜스포머 변형 모형에 대해 조사하고 정리하였다.
Abstract
Pre-trained language models are the most important and widely used tools in natural language processing tasks. Since those have been pre-trained for a large amount of corpus, high performance can be expected even with fine-tuning learning using a small number of data. Since the elements necessary for implementation, such as a pre-trained tokenizer and a deep learning model including pre-trained weights, are distributed together, the cost and period of natural language processing has been greatly reduced. Transformer variants are the most representative pre-trained language models that provide these advantages. Those are being actively used in other fields such as computer vision and audio applications. In order to make it easier for researchers to understand the pre-trained language model and apply it to natural language processing tasks, this paper describes the definition of the language model and the pre-learning language model, and discusses the development process of the pre-trained language model and especially representative Transformer variants.
참고문헌 (33)
- T. Lin, "A survey of transformers", AI Open, ~ (2022)
- A. Dosovitskiy, "An image is worth 16x16words: Transformers for image recognition at scale", arXiv preprint arXiv:2010.11929, ~ (2020)
- M. Chen, "Generative pretraining from pixels", International conference on machine learning. PMLR (2020)
- C. Subakan, "Attention is all you need in speech separation", ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2021), IEEE
- H. Akbari, "Vatt : Transformers for multimodal self-supervised learning from raw video, audio and text", Advances in Neural Information Processing Systems, 34, pp.24206~24221 (2021)
- H. Li, "Language models : past, present, and future", Communications of the ACM, 65(7), pp.56~63 (2022)
- A. Radford, "Language models are unsupervised multitask learners", OpenAI blog, 1(8), pp.9~ (2019)
- J. Devlin, "Bert : Pre-training of deep bidirectional transformers for language understanding", arXiv preprint arXiv:1810.04805, ~ (2018)
- T. Brown, "Language models are few-shot learners", Advances in neural information processing systems, 33, pp.1877~1901 (2020)
- S.J. P an, "A survey o n tran sfer learning", IEEE Transactions on knowledge and data engineering, 22(10), pp.1345~1359 (2009)
- Y. Bengio, "A neural probabilistic language model", Advances in neural information processing systems, 13 (2000)
- T. Mikolov, "Efficient estimation of word representations in vector space", arXiv preprint arXiv:1301.3781, ~ (2013)
- J. Sarzynska-Wawer, "Detecting formal thought disorder by deep contextualized word representations", Psychiatry Research, 304, pp.114135~ (2021)
- A. Vaswani, "Attention is all you need", Advances in neural information processing systems, 30, pp.5998~6008 (2017)
- J.L. Ba, "Layer normalization", arXiv preprint arXiv:1607.06450, ~ (2016)
- Y. Wu, "Google’s neural machine translation system : Bridging the gap between human and machine translation", arXiv preprint arXiv:1609. 08144, ~ (2016)
- A. Radford, "Improving language understanding by generative pre-training", ~ (2018)
- A. Wang, "GLUE : A multi-task benchmark and analysis platform for natural language understanding", arXiv preprint arXiv:1804.07461, ~ (2018)
- A. Wang, "Superglue : A stickier benchmark for general-purpose language understanding systems", Advances in neural information processing systems, 32 (2019)
- Y. Liu, "Roberta : A robustly optimized bert pretraining approach", arXiv preprint arXiv:1907. 11692, ~ (2019)
- Z. Lan, "Albert : A lite bert for self-supervised learning of language representations", arXiv preprint arXiv:1909.11942, ~ (2019)
- K. Clark, "Electra : Pre-training text encoders as discriminators rather than generators", arXiv preprint arXiv:2003.10555, ~ (2020)
- M. Lewis, "Bart : Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension", arXiv preprint arXiv:1910.13461, ~ (2019)
- S. Lee, "Kr-bert: A small-scale korean-specific language model", arXiv preprint arXiv:2008.03979, ~ (2020)