LLM(Large Language Model) 속성과 성능 연관성 연구
A study on the Correlation Between LLM (Large Language Model) Properties and Performance
0
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
OpenAI의 ChatGPT는 기존 기술보다 놀랍게 향상된 기능을 보여주었고 생성형 AI에 대한 활발한 연구개발과 활용이 시작되었다. 컴퓨터 성능의 비약적인 발전으로 거대한 크기의 인공지능 언어모델(LLM)을 활용하게 되었다. 본 연구는 LLM을 구축할 때 성능을 평가할 방법과 성능을 높이기 위해 LLM의 구축 모든 과정에 걸쳐 고려해야 할 요인을 제시한다. 원시 데이터로부터 학습하여 backbone 모델을 만드는 것부터 만들어진 모델에 fine-tuning을 통해서 최종 모델을 구축하는 과정을 포함한다. 학습 데이터의 양과 품질, 모델 및 파라미터크기, 학습횟수 및 fine-tuning에 관련된 방안을 제시한다. 영문모델은 크기를 줄이면서 성능을 높이는 성과가 있는 반면에 한국어 모델은 작고 성능이 우수하면서 연구개발과 서비스에 제한 없이 활용 가능한 오픈소스 모델의 구축과 보급이 필수적이다.
Abstract
OpenAI's ChatGPT showed remarkable improvements over existing technologies, and active research and development and utilization of generative AI began. Advances in computing power have led to the use of large-scale artificial intelligence language models (LLMs). This study presents methods to evaluate performance when building an LLM and factors to be considered throughout the entire process of building an LLM to increase performance. It includes the entire process of building a model by learning from raw data and fine-tuning the created backbone model. We present how to improve performance from considering the quantity and quality of learning data, model and parameter size, learning frequency, and fine tuning. When it comes to the English dataset, there have been achievements in improving performance while reducing the model size, while in the case of Korean, it is essential to build and disseminate an open source model that is small but has excellent performance and can be used without restrictions for research and development and services.
참고문헌 (39)
- M. Shanahan, "Talking about large language models", CoRR, abs, pp.2212.03551 (2022), ~
- Jurafsky, Daniel, "N-gram language models", Speech and language processing, 23 (2018)
- Wang, Yu-An, "What do position embeddings learn? an empirical study of pre-trained language model positional encoding", ~ (2020)
- Vaswani, Ashish, "Attention is all you need", Advances in neural information processing systems (2013)
- Koroteev, M. V., "BERT : a review of applications in natural language processing and understanding", ~ (2021)
- Lewis, Mike, "Bart : Denoising sequence-tosequence pre-training for natural language generation, translation, and comprehension", ~ (2019)
- Touvron, Hugo, "Llama : Open and efficient foundation language models", ~ (2023)
- West, "Spring research presentation: A theoretical foundation for inductive transfer", Brigham Young University, College of Physical and Mathematical Sciences, 1(08) (2007)
- Friederich, Simon, "Fine-tuning", 2017, ~
- Renggli, Cedric, "SHiFT : an efficient, flexible search engine for transfer learning", ~ (2022)
- Brown, Tom, "Language models are few-shot learners", Advances in neural information processing systems, ~ (2020)
- Jaiyam Sharma, "Understanding Attention Mechanism in Transformer Neural Networks", LearnOpenCV, ~ (2023)
- Hu, Edward J., "Lora : Low-rank adaptation of large language models", ~ (2021)
- Gao, Leo, "The pile : An 800gb dataset of diverse text for language modeling", ~ (2020)
- Ko, Hyunwoong, "A Technical Report for Polyglot-Ko : Open-Source Large-Scale Korean Language Models", ~ (2023)
- Sennrich, Rico, "Neural machine translation of rare words with subword units", ~ (2015)
- Schuster, Mike, "Japanese and korean voice search", 2012 IEEE international conference on acoustics, speech and signal processing (ICASSP) (2012), IEEE
- Kudo, Taku,, "Sentencepiece : A simple and language independent subword tokenizer and detokenizer for neural text processing", ~ (2018)
- Ullah, Fahad, "A self-attention model for inferring cooperativity between regulatory features", Nucleic acids research, 49(13) (2021)
- Bengio, Yoshua, "A neural probabilistic language model", Advances in neural information processing systems (2000)
- Lin, Stephanie, "Truthfulqa : Measuring how models mimic human falsehoods", ~ (2021)
- Zellers, Rowan, "Hellaswag : Can a machine really finish your sentence?", ~ (2019)
- Clark, Peter, "Think you have solved question answering? try arc, the ai2 reasoning challenge", ~ (2018)
- E. Frantar, "Gptq : Accurate post-training quantization for generative pre-trained transformers", ~ (2022)
- Paperno, Denis, "The LAMBADA dataset : Word prediction requiring a broad discourse context", ~ (2016)
- Joshi, Mandar, "Triviaqa : A large scale distantly supervised challenge dataset for reading comprehension", ~ (2017)
- Rajpurkar, Pranav, "Squad : 100, 000+ questions for machine comprehension of text", ~ (2016)
- J. Huang, "Towards reasoningin large language models : A survey", CoRR, abs, pp.2212.10403 (2022), ~
- Bisk, Yonatan, "Piqa : Reasoning about physical commonsense in natural language", Proceedings of the AAAI conference on artificial intelligence, 34(05) (2020)
- R. Taylor, "Galactica : A large language model for science", CoRR, abs, pp.2211. 09085 (2022), ~
- J. Hoffmann, "Training computeoptimal large language models", abs, pp.2203. 15556 (2022), ~
- Song, Xinying, "Fast wordpiece tokenization", ~ (2020)
- Rae, Jack W., "Scaling language models : Methods, analysis & insights from training gopher", ~ (2021)
- "Loss Functions"[URL]
- R. Taori, "Stanford Alpaca: An Instruction-following LLaMA Model" (2023), github
- The Vicuna Team, "Vicuna : An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality", LMSYSORG, ~ (2023)