생성 모델과 검색 모델을 이용한 한국어 멀티턴 응답 생성 연구
A study on Korean multi-turn response generation using generative and retrieval model
1
조회
0
다운로드
3
인용
0
참고문헌
키워드
초록
최근 딥러닝 기반의 자연어처리 연구는 사전 훈련된 언어 모델을 통해 대부분의 자연어처리 분야에서 우수한 성능을 보인다. 특히 오토인코더 (auto-encoder) 기반의 언어 모델은 다양한 한국어 이해 분야에서 뛰어난 성능과 쓰임을 증명하고 있다. 그러나 여전히 디코더 (decoder) 기반의 한국어 생성 모델은 간단한 문장 생성 과제에도 어려움을 겪고 있으며, 생성 모델이 가장 일반적으로 쓰이는 대화 분야에서의 세부 연구와 학습 가능한 데이터가 부족한 상황이다. 따라서 본 논문은 한국어 생성 모델을 위한 멀티턴 대화 데이터를 구축하고 전이 학습을 통해 생성 모델의 대화 능력을 개선하여 성능을 비교 분석한다. 또한, 검색 모델을 통해 외부 지식 정보에서 추천 응답 후보군을 추출하여 모델의 부족한 대화 생성 능력을 보완하는 방법을 제안한다.
Abstract
Recent deep learning-based research shows excellent performance in most natural language processing (NLP) fields with pre-trained language models. In particular, the auto-encoder-based language model proves its excellent performance and usefulness in various fields of Korean language understanding. However, the decoder-based Korean generative model even suffers from generating simple sentences. Also, there is few detailed research and data for the field of conversation where generative models are most commonly utilized. Therefore, this paper constructs multi-turn dialogue data for a Korean generative model. In addition, we compare and analyze the performance by improving the dialogue ability of the generative model through transfer learning. In addition, we propose a method of supplementing the insufficient dialogue generation ability of the model by extracting recommended response candidates from external knowledge information through a retrival model.
참고문헌 (25)
- A. Vaswani, "Attention is all you need", Advances in neural information processing systems, pp.5998~6008 (2017)
- J.A. Bernard, "Use of a rule-based system for process control", IEEE Control Systems Magazine, 8(5), pp.3~13 (1988), Institute of Electrical and Electronics Engineers (IEEE)[DOI]
- C. Manning, "Foundations of statistical natural language processing" (1999), MIT press
- J. Devlin, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, pp.4171~4186 (2019)
- Y. Liu, "Roberta: A robustly optimized bert pretraining approach", arXiv preprint arXiv:1907.11692, ~ (2019)
- K. Clark, "ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators", International Conference on Learning Representations (2020)
- S. Park, "KLUE:Korean Language Understanding Evaluation", arXiv preprint arXiv:2105.09680, ~ (2021)
- A. Radford, "Language models are unsupervised multitask learners", OpenAI blog, 1(8), pp.9~ (2019)
- T. B. Brown, "Language models are few-shot learners", arXiv preprint arXiv:2005.14165, ~ (2020)
- M. Lewis, "Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension", arXiv preprint arXiv:1910.13461, ~ (2019)
- C. Park, "TS: Back TranScription for speech-to-text post-processor using text-to-speech-to-text", Proceedings of the 8th Workshop on Asian Translation (WAT2021), pp.106~116 (2021)
- K. Gregor, "Deep autoregressive networks", International Conference on Machine Learning, pp.1242~1250 (2014), PMLR
- H. Rashkin, "Towards empathetic open-domain conversation models: A new benchmark and dataset", arXiv preprint arXiv:1811.00207, ~ (2018)
- Stephen Robertson, "The Probabilistic Relevance Framework: BM25 and Beyond", Foundations and Trends® in Information Retrieval, 3(4), pp.333~389 (2009), Now Publishers[DOI]
- "Poly-encoders:Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring. In International Conference on Learning Representations" (2019)[URL]
- T. Wolf, "Huggingface's transformers: State-of-the-art natural language processing", arXiv preprint arXiv:1910.03771, ~ (2019)
- I. Loshchilov, "Decoupled weight decay regularization", arXiv preprint arXiv:1711.05101, ~ (2017)
- S. Wiseman, "Sequence-to-Sequence Learning as Beam-Search Optimization", Proceedings of the 2016Conference on Empirical Methods in Natural Language Processing, pp.1296~1306 (2016)
- S. Roller, "Recipes for building an open-domain chatbot", arXiv preprint arXiv:2004.13637, ~ (2020)
- K. Papineni, "Bleu: a method for automatic evaluation of machine translation", Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp.311~318 (2002)
- "Rouge: A package for automatic evaluation of summaries. In Text summarization branches out" (2004)[URL]
- "BERTScore:Evaluating Text Generation with BERT. In International Conference on Learning Representations" (2019)[URL]
- S. Gehrmann, "The gem benchmark : Natural language generation, its evaluation and metrics", arXiv preprint arXiv:2102.01672, ~ (2021)
- T. Schick, "It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners", Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies, pp.2339~2352 (1815)
- T. Gao, "Making pre-trained language models better few-shot learners", arXiv preprint arXiv:2012.15723, ~ (2020)