텍스트 요약 품질 향상을 위한 의미적 사전학습 방법론
Semantic Pre-training Methodology for Improving Text Summarization Quality
0
조회
0
다운로드
4
인용
0
참고문헌
키워드
초록
최근 사용자에게 의미있는 정보만을 자동으로 간추리는 텍스트 자동 요약이 꾸준히 연구되고 있으며, 특히 인공신경망 모델인 트랜스포머를 활용한 텍스트 요약 연구가 주로 수행되고 있다. 다양한 연구 중 특히 문장 단위 마스킹을 통해 모델을 학습시키는 GSG 방식이 가장 주목을 받고 있지만, 전통적인 GSG는 문장의 의미가 아닌 토큰의 중복 정도에 기반을 두어 마스킹 대상 문장을 선정한다는 한계를 갖는다. 따라서 본 연구에서는 텍스트 요약의 품질을 향상시키기 위해, 문장의 의미를 고려하여 GSG의 마스킹 대상 문장을 선정하는 SbGSG(Semantic-based GSG) 방법론을 제안한다. 뉴스기사 370,000건과 요약문 및 레포트 21,600건을 사용하여 실험을 수행한 결과, ROUGE와 BERT Score 측면에서 제안 방법론인 SbGSG가 전통적인 GSG에 비해 우수한 성능을 보임을 확인하였다.
Abstract
Recently, automatic text summarization, which automatically summarizes only meaningful information for users, is being studied steadily. Especially, research on text summarization using Transformer, an artificial neural network model, has been mainly conducted. Among various studies, the GSG method, which trains a model through sentence-by-sentence masking, has received the most attention. However, the traditional GSG has limitations in selecting a sentence to be masked based on the degree of overlap of tokens, not the meaning of a sentence. Therefore, in this study, in order to improve the quality of text summarization, we propose SbGSG (Semantic-based GSG) methodology that selects sentences to be masked by GSG considering the meaning of sentences. As a result of conducting an experiment using 370,000 news articles and 21,600 summaries and reports, it was confirmed that the proposed methodology, SbGSG, showed superior performance compared to the traditional GSG in terms of ROUGE and BERT Score.
참고문헌 (31)
- Y. Liu, "Fine-tune BERT for extractive summarization" (2019)
- J. Xu, "Neural extractive text summarization with syntactic compression" (2019)
- M. Zhong, "Extractive summarization as text matching" (2020)
- R. Nallapati, "Abstractive text summarization using sequence-to-sequence rnns and beyond" (2016)
- A. M. Rush, "A neural attention model for abstractive sentence summarization" (2015)
- A. See, "Get to the point: Summarization with pointer-generator networks" (2017)
- A. Vaswani, "Attention is all you need", Advances in Neural Information Processing Systems, 30 (2017)
- J. Zhang, "Pegasus: Pre-training with extracted gap-sentences for abstractive summarization", Proceedings of the 37th International Conference on Machine Learning, 119, pp.11328~11339 (2020)
- C. Lin, "Rouge : A package for automatic evaluation of summaries", Text Summarization Branches Out, pp.74~81 (2004)
- H. P. Luhn, "A statistical approach to mechanized encoding and searching of literary information", IBM Journal of Research and Development, 1(4), pp.309~317 (1957)
- M. A. Fattah, "GA, MR, FFNN, PNN and GMM based models for automatic text summarization", Comput. Speech Lang, 23(1), pp.126~144 (2009)
- R. Mihalcea, "Textrank: Bringing order into text", Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pp.404~411 (2004)
- The PageRank Citation Ranking: Bringing Order to the Web, "L. Page", Stanford University, ~ (1998)
- 차준석, "단어 간 의미적 연관성을 고려한 어휘 체인 기반의 개선된 자동 문서요약 방법", 스마트미디어저널, 6(1), pp.22~29 (2017), (사)한국스마트미디어학회
- R. Nallapati, "Classify or select : Neural architectures for extractive document summarization" (2016)
- A. Khan, "A review on abstractive summarization methods", Journal of Theoretical and Applied Information Technology, 59(1), pp.64~72 (2014)
- 이태석, "미등록 어휘에 대한 선택적 복사를 적용한 문서 자동요약", 스마트미디어저널, 8(2), pp.58~65 (2019), (사)한국스마트미디어학회[DOI]
- S. Hochreiter, "Long short-term memory", Neural Comput, 9(8), pp.1735~1780 (1997)
- I. Sutskever, "Sequence to sequence learning with neural networks", Advances in Neural Information Processing Systems, 27 (2014)
- D. Bahdanau, "Neural machine translation by jointly learning to align and translate" (2014)
- A. Radford, "Improving language understanding by generative pre-training", ~ (2018)
- A. Radford, "Language models are unsupervised multitask learners", OpenAI Blog, 1(8), pp.9~ (2019)
- T. Brown, "Language models are few-shot learners", Advances in Neural Information Processing Systems, 33, pp.1877~1901 (2020)
- R. Thoppilan, "Lamda: Language models for dialog applications" (2022)
- H. Touvron, "Llama: Open and efficient foundation language models" (2023)
- A. Chowdhery, "Palm : Scaling language modeling with pathways" (2022)
- J. Devlin, "Bert: Pre-training of deep bidirectional transformers for language understanding" (2018)
- M. Joshi, "Spanbert : Improving pre-training by representing and predicting spans", Transactions of the Association for Computational Linguistics, 8, pp.64~77 (2020)
- 김은희, "ELMo 임베딩 기반 문장 중요도를 고려한 중심 문장 추출방법", 스마트미디어저널, 10(1), pp.39~46 (2021), (사)한국스마트미디어학회[DOI]
- N. Reimers, "Sentence-bert : Sentence embeddings using siamese bert-networks" (2019)
- C. Raffel, "Exploring the limits of transfer learning with a unified text-to-text transformer", The Journal of Machine Learning Research, 21(1), pp.5485~5551 (2020)