대형 사전학습 언어모델 연구에 대한 고찰
A consideration on research for large scale pretrained language model
0
조회
0
다운로드
4
인용
0
참고문헌
키워드
초록
The pre-training language model(PLM), BERT has achieved great success in Natural Language Processing(NLP) by transferring knowledge from rich-resource pre-training task to low-resource downstream tasks. The focus of researchers’ attention on PLM is its application and performance. In this study, we were interested in large-scale PLM models with high performance. We tried to identify major issues through PLM-related researches, and to discuss on the present development situation of large-scale PLMs. First, we reviewed researches analyzing whether PLM acquires linguistic/world knowledge through pre-training task. PLM remains as a blackbox and researchers fail to identify the internal mechanism of PLM. The aim of a large-scale PLM is to make the model more general-purposed. The larger the number of parameter is, the more difficult to train the parameters. Instead of fine-tuning method, few-shot method was proposed, but further study remains to be done. The final question is whether making larger PLM is the right research direction. Due to the overconsumption of computing and energy resources by PLM, we propose the development of eco-friendly and explainable PLM. We hope this study will provide linguists with an opportunity to understand recent technological research trends and to find joint work in the field of NLP.
Abstract
The pre-training language model(PLM), BERT has achieved great success in Natural Language Processing(NLP) by transferring knowledge from rich-resource pre-training task to low-resource downstream tasks. The focus of researchers’ attention on PLM is its application and performance. In this study, we were interested in large-scale PLM models with high performance. We tried to identify major issues through PLM-related researches, and to discuss on the present development situation of large-scale PLMs. First, we reviewed researches analyzing whether PLM acquires linguistic/world knowledge through pre-training task. PLM remains as a blackbox and researchers fail to identify the internal mechanism of PLM. The aim of a large-scale PLM is to make the model more general-purposed. The larger the number of parameter is, the more difficult to train the parameters. Instead of fine-tuning method, few-shot method was proposed, but further study remains to be done. The final question is whether making larger PLM is the right research direction. Due to the overconsumption of computing and energy resources by PLM, we propose the development of eco-friendly and explainable PLM. We hope this study will provide linguists with an opportunity to understand recent technological research trends and to find joint work in the field of NLP.
참고문헌 (24)
- 임준호, "딥러닝 사전학습 언어모델 연구 동향", 정보과학회지, 39(4), pp.9~14 (2021)
- 지인영, "사전학습 언어모델의 기술 분석 연구", 인문언어, 22(1), pp.111~133 (2020), 국제언어인문학회[DOI]
- 지인영, "사전학습 언어모델과 언어학정보", 2021년 담화인지언어학회 춘계학술대회논문집, pp.175~190 (2021)
- Belinkov, Yonatan, "Probing classifiers: Promises, shortcomings, and alternatives", arXiv:2102.12452v2, ~ (2021)
- Bender, Emily M., "On the dangers of stochastic parrots: Can language models be too big?", FAccT '21: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (2021)
- Brown, Tom B., "Language models are few-shot learners", arXiv:2005.14165v4, ~ (2020)
- Chen, Tianlong, "The lottery ticket hypothesis for pre-trained BERT networks", NeurIPS 2020 (2020)
- Devlin, Jacob, "Bert:Pre-training of deep bidirectional transformers for language understanding", arXiv:1810.04805v2, ~ (2018)
- Frankle, Jonathan, "The lottery ticket hypothesis: Finding sparse, trainable neural networks", ICLR 2019 (2018)
- Han, Xu, "Pre-trained models: Past, present and future", arXiv:2106.07139v3, ~ (2021)
- Kim, Boseop, "What changes can large-scale language models bring? Intensive study on HyperCLOVA: Billions-scale Korean generative pretrained transformers", arXiv:2109.04650v1 EMNLP 2021, ~ (2021)
- "2019", ~
- Tal Linzen, "Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies", Transactions of the Association for Computational Linguistics, 4, pp.521~535 (2016), MIT Press - Journals[DOI]
- Liu, Yinhan, "RoBERTa: A robustly optimized BERT pretraining approach", arXiv:1907.11692v, ~ (2019)
- Park, Sungjoon, "KLUE: Korean language understanding evaluation", arXiv:2105.09680v4, ~ (2021)
- "Improving language understanding by generative pretraining. openAI GPT" (2018)[URL]
- "Language models are unsupervised multitask learners. Open AI GPT2" (2019)[URL]
- Raffel, Colin, "Exploring the limits of transfer learning with a unified text-to-text transformer", arXiv:1910.10683v3, ~ (2019)
- "2020", ~
- "Turing-NLG: A 17-billion parameter language model by microsoft" (2020)[URL]
- Shoeyb, Mohammad, "Megatron-LM: Training multi-billion parameter language models using model parallelism", arXiv:1909.08053v4, ~ (2019)
- Vaswani, Ashish, "Attention is all you need", NIPS 2017 (2017)
- Warstadt, Alex, "Investigating BERT’s knowledge of language: Five analysis methods with NPIs", arXiv:1909.02597v2 [cs.CL, ~ (2019)
- Zeng, Wei, "PanGu-α: Large-scale autoregressive pretrained Chinese language models with auto-parallel computation", arXiv:2104.12369v1, ~ (2021)