Improving Abstractive Summarization by Training Masked Out-of-Vocabulary Words
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Text summarization is the task of producing a shorter version of a long document while accurately preservingthe main contents of the original text. Abstractive summarization generates novel words and phrases using alanguage generation method through text transformation and prior-embedded word information. However,newly coined words or out-of-vocabulary words decrease the performance of automatic summarization becausethey are not pre-trained in the machine learning process. In this study, we demonstrated an improvement insummarization quality through the contextualized embedding of BERT with out-of-vocabulary masking. Inaddition, explicitly providing precise pointing and an optional copy instruction along with BERT embedding,we achieved an increased accuracy than the baseline model. The recall-based word-generation metric ROUGE-1 score was 55.11 and the word-order-based ROUGE-L score was 39.65.
Abstract
Text summarization is the task of producing a shorter version of a long document while accurately preservingthe main contents of the original text. Abstractive summarization generates novel words and phrases using alanguage generation method through text transformation and prior-embedded word information. However,newly coined words or out-of-vocabulary words decrease the performance of automatic summarization becausethey are not pre-trained in the machine learning process. In this study, we demonstrated an improvement insummarization quality through the contextualized embedding of BERT with out-of-vocabulary masking. Inaddition, explicitly providing precise pointing and an optional copy instruction along with BERT embedding,we achieved an increased accuracy than the baseline model. The recall-based word-generation metric ROUGE-1 score was 55.11 and the word-order-based ROUGE-L score was 39.65.
참고문헌 (26)
- "Text summarization techniques: a brief survey" (2017)[URL]
- N. Nazari, "A survey on automatic text summarization", Journal of AI and Data Mining, 7(1), pp.121~135 (2019)
- "Ranking sentences for extractive summarization with reinforcement learning" (2018)[URL]
- J. Cheng, "Neural summarization by extracting sentences and words", Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pp.484~494 (2016)
- W. Wang, "Concept pointer network for abstractive summarization", Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp.3076~3085 (2019)
- G. Rossiello, "Centroid-based text summarization through compositionality of word embeddings", Proceedings of the MultiLing 2017 Workshop on Summarization and Summary Evaluation Across Source Types and Genres, pp.12~21 (2017)
- "BERT: pre-training of deep bidirectional transformers for language understanding" (2018)[URL]
- "A survey on neural network-based summarization methods" (2018)[URL]
- Tian Shi, "Neural Abstractive Text Summarization with Sequence-to-Sequence Models", ACM, 2021, pp.2(1), 1~37[DOI]
- "Attention-guided answer distillation for machine reading comprehension" (2018)[URL]
- I. Sutskever, "Sequence to sequence learning with neural networks", Advances in Neural Information Processing Systems, 27, pp.3104~3112 (2014)
- A. Vaswani, "Attention is all you need", Advances in Neural Information Processing Systems, 30, pp.5998~6008 (2017)
- "Neural machine translation by jointly learning to align and translate" (2014)[URL]
- S. Xu, "Self-attention guided copy mechanism for abstractive summarization", Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pp.1355~1362 (2020)
- J. Gu, "Incorporating copying mechanism in sequence-to-sequence learning", Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pp.1631~1640 (2016)
- C. Gulcehre, "Pointing the unknown words", Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pp.140~149 (2016)
- "Abstractive text summarization using sequence-tosequence RNNs and beyond" (2016)[URL]
- "Rouge 2.0: updated and improved measures for evaluation of summarization tasks" (2018)[URL]
- M. Yasunaga, "ScisummNet : a large annotated corpus and content-impact models for scientific paper summarization with citation networks", Proceedings of the AAAI Conference on Artificial Intelligence, pp.7386~7393 (2019)
- S. Erera, "A summarization system for scientific documents", Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP), pp.211~216 (2019)
- 이태석, "BERT 임베딩과 선택적 OOV 복사 방법을 사용한 문서요약", 정보과학회논문지, 47(1), pp.36~44 (2020), 한국정보과학회[DOI]
- L. N. Smith, "Cyclical learning rates for training neural networks", Proceedings of 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp.464~472 (2017)
- "SGDR: stochastic gradient descent with warm restarts" (2016)[URL]
- J. Howard, "Universal language model fine-tuning for text classification", Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pp.328~339 (2018)
- "Snapshot ensembles: train 1, get m for free" (2017)[URL]
- K. Goyal, "A continuous relaxation of beam search for end-toend training of neural sequence models", Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pp.3045~3052 (2018)