오류 유형에 따른 생성요약 모델의 본문-요약문 간 요약 성능평가 비교
Empirical Study for Automatic Evaluation of Abstractive Summarization by Error-Types
1
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
텍스트 생성요약은 자연어처리의 과업 중 하나로 긴 텍스트의 내용을 보존하면서 짧게 축약된 요약문을 생성한다. 생성요약 과업의 특성 상 본문의 핵심내용을 요약문에서 보존하는 것은 매우 중요하다. 기존의 생성요약 방법론은 정답요약과의 어휘 중첩도(Lexical-Overlap)를 기반으로 본문의 내용과 유창성을 측정했다. ROUGE는 생성요약 요약모델의 평가지표로 많이 사용하는 어휘 중첩도 기반의 평가지표이다. 생성요약 벤치마크에서 ROUGE가 49점대로 매우 높은 성능을 보임에도 불구하고, 생성한 요약문과 본문의 내용이 불일치하는 경우가 30% 가량 존재한다. 본 연구에서는 정답요약의 도움 없이 본문만을 활용해 생성요약 모델의 성능을 평가하는 방법론을 제안한다. 본 연구에서 제안한 평가점수를 AggreFACT의 라벨과 상관도 분석결과, 다음의 두 가지 경우 가장 높은 상관관계를 보였다. 첫 번째는 Transformer 구조의 인코더-디코더 구조에 대규모 사전학습을 진행한 BART와 PEGASUS 등을 생성요약 모델의 베이스라인으로 사용한 경우이고, 두 번째는 요약문 전체에 걸쳐 오류가 발생한 경우이다.
Abstract
Generative Text Summarization is one of the Natural Language Processing tasks. It generates a short abbreviated summary while preserving the content of the long text. ROUGE is a widely used lexical-overlap based metric for text summarization models in generative summarization benchmarks. Although it shows very high performance, the studies report that 30% of the generated summary and the text are still inconsistent. This paper proposes a methodology for evaluating the performance of the summary model without using the correct summary. AggreFACT is a human-annotated dataset that classifies the types of errors in neural text summarization models. Among all the test candidates, the two cases, generation summary, and when errors occurred throughout the summary showed the highest correlation results. We observed that the proposed evaluation score showed a high correlation with models finetuned with BART and PEGASUS, which is pretrained with a large-scale Transformer structure.
참고문헌 (73)
- Amplayo, R. K., "Smart : Sentences as basic units for text evaluation", arXiv preprint arXiv:2208.01030, ~ (2022)
- Arumae, K., "Guiding extractive summarization with question-answering rewards", Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: Human language technologies, 1, pp.2566~2577 (2019), Association for Computational Linguistics
- Beltagy, I., "Longformer : The long-document transformer", arXiv: 2004.05150, ~ (2020)
- Berry, M. W., "Using linear algebra for intelligent information retrieval", SIAM Review, 37(4), pp.573~595 (1995)[DOI]
- Cachola, I., "Findings of the association for computational linguistics: Emnlp 2020", pp.4766~4777 (2020), Association for Computational Linguistics
- Cao, S., "Cliff : Contrastive learning for improving faithfulness and factuality in abstractive summarization", arXiv preprint arXiv:2109.09209, ~ (2021)
- Cao, Z., "Faithful to the original: Fact-aware neural abstractive summarization", Proceedings of the thirty-second aaai conference on artificial intelligence and thirtieth innovative applications of artificial intelligence conference and eighth aaai symposium on educational advances in artificial intelligence (2018), AAAI Press
- Cohan, A., "A discourse-aware attention model for abstractive summarization of long documents", Proceedings of the 2018 conference of the north American chapter of the association for computational linguistics: Human language technologies, 2, pp.615~621 (2018), Association for Computational Linguistics
- Devlin, J., "Bert : Pretraining of deep bidirectional transformers for language understanding", arXiv preprint arXiv:1810.04805, ~ (2018)
- Durmus, E., "FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization", Proceedings of the 58th annual meeting of the association for computational linguistics, pp.5055~5070 (2020), Association for Computational Linguistics
- Erkan, G., "Lexrank : Graph-based lexical centrality as salience in text summarization", Journal of artificial intelligence research, 22, pp.457~479 (2004)
- Fabbri, A., "QAFactEval: Improved QA-based factual consistency evaluation for summarization", Proceedings of the 2022 conference of the north american chapter of the association for computational linguistics: Human language technologies, pp.2587~2601 (2022), Association for Computational Linguistics
- Falke, T., "Ranking generated summaries by correctness: An interesting but challenging application for natural language inference", Proceedings of the 57th annual meeting of the association for computational linguistics, pp.2214~2220 (2019), Association for Computational Linguistics
- Gabriel, S., "GO FIGURE: A meta evaluation of factuality in summarization", Findings of the association for computational linguistics: Acl-ijcnlp 2021, pp.478~487 (2021), Association for Computational Linguistics
- Gliwa, B., "SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization", Proceedings of the 2nd workshop on new frontiers in summarization, pp.70~79 (2019), Association for Computational Linguistics
- Goodfellow, I., "Deep learning(Vol. 1)" (2016), MIT Press
- Goyal, T., "Annotating and modeling fine-grained factuality in summarization", Proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: Human language technologies (2021)
- Gudivada, V. N., "Computational analysis and understanding of natural languages: Principles, methods and applications (Vol. 38)", pp.403~428 (2018), Elsevier
- Gupta, P., "Dialfact : A benchmark for fact-checking in dialogue", arXiv preprint arXiv:2110.08222, ~ (2021)
- Hardy, Narayan, S., "Highres: Highlight-based referenceless evaluation of summarization" (2019)
- Hermann, K. M., "Teaching machines to read and comprehend", Advances in neural information processing systems, 28 (2015)
- Honovich, O., "True: Re-evaluating factual consistency evaluation", Workshop on document-grounded dialogue and conversational question answering (2022)
- Huang, L., "Efficient attentions for long document summarization", Proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: Human language technologies, pp.1419~1436 (2021), Association for Computational Linguistics
- Ji, Z., "Survey of hallucination in natural language generation", ACM Comput. Surv., 55(12) (2023)[DOI]
- Kågebäck, M., "Extractive summarization using continuous vector space models", Proceedings of the 2nd workshop on continuous vector space models and their compositionality (cvsc), pp.31~39 (2014)
- Kim, B., "Abstractive summarization of Reddit posts with multi-level memory networks", Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: Human language technologies, 1, pp.2519~2531 (2019), Association for Computational Linguistics
- Kim, D. -H., "Query-based document summarization using important sentence selection heuristics and mmr", Annual conference on human and language technology, pp.285~291 (2002)
- Koto, F., "Ffci: A framework for interpretable automatic evaluation of summarization", J. Artif. Intell. Res., 73 (2020)
- Kryscinski, W., "Evaluating the factual consistency of abstractive text summarization", Proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pp.9332~9346 (2020), Association for Computational Linguistics
- Philippe Laban, "<scp>SummaC<", scp>: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization, 2022, pp.10, 163~177[DOI]
- Ladhak, F., "WikiLingua : A new benchmark dataset for cross-lingual abstractive summarization", Findings of the association for computational linguistics: Emnlp 2020, pp.4034~4048 (2020), Association for Computational Linguistics
- Lavie, A., "METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments", Proceedings of the second workshop on statistical machine translation, pp.228~231 (2007), Association for Computational Linguistics
- Lewis, M., "BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension", Proceedings of the 58th annual meeting of the association for computational linguistics, pp.7871~7880 (2020)
- Lin, C.-Y., "Text summarization branches out", pp.74~81 (2004), Association for Computational Linguistics
- Liu, Y., "Fine-tune bert for extractive summarization", arXiv preprint arXiv : 1903, ~ (2019)
- Liu, Y., "Text summarization with pretrained encoders", arXiv preprint arXiv: 1908.08345, ~ (2019)
- Liu, Y., "Simcls : A simple framework for contrastive learning of abstractive summarization", arXiv preprint arXiv:2106.01890, ~ (2021)
- Liu, Y., "Brio: Bringing order to abstractive summarization", arXiv preprint arXiv:2203.16804, ~ (2022)
- Maynez, J., "On faithfulness and factuality in abstractive summarization", Proceedings of the 58th annual meeting of the association for computational linguistics, pp.1906~1919 (2020), Association for Computational Linguistics
- Mihalcea, R., "Textrank: Bringing order into text", Proceedings of the 2004 conference on empirical methods in natural language processing, pp.404~411 (2004)
- Narayan, S., "Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization", arXiv preprint arXiv: 1808.08745, ~ (2018)
- Pagnoni, A., "Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics", Proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: Human language technologies, pp.4812~4829 (2021), Association for Computational Linguistics
- Papineni, K., "Bleu: a method for automatic evaluation of machine translation", Proceedings of the 40th annual meeting of the association for computational linguistics, pp.311~318 (2002), Association for Computational Linguistics
- Pasunuru, R., "Multi-reward reinforced summarization with saliency and entailment", Proceedings of the 2018 conference of the north American chapter of the association for computational linguistics: Human language technologies, 2, pp.646~653 (2018), Association for Computational Linguistics
- Paulus, R., "A deep reinforced model for abstractive summarization", ArXiv, abs, 2017
- Raffel, C., "Exploring the limits of transfer learning with a unified textto-text transformer", Journal of Machine Learning Research, 21(140), pp.1~67 (2020)[DOI]
- Rajpurkar, P., "SQuAD: 100,000+ questions for machine comprehension of text", Proceedings of the 2016 conference on empirical methods in natural language processing, pp.2383~2392 (2016), Association for Computational Linguistics
- Sascha Rothe, "Leveraging Pre-trained Checkpoints for Sequence Generation Tasks", Transactions of the Association for Computational Linguistics, 8, pp.264~280 (2020), MIT Press - Journals[DOI]
- "Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter" (2019)[URL]
- Schuster, T., "Get your vitamin C! robust fact verification with contrastive evidence", Proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: Human language technologies, pp.624~643 (2021), Association for Computational Linguistics
- Scialom, T., "MLSUM : The multilingual summarization corpus", Proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pp.8051~8067 (2020), Association for Computational Linguistics
- Scialom, T., "Questeval : Summarization asks for fact-based evaluation", arXiv preprint arXiv:2103.12693, ~ (2021)
- Scialom, T., "Answers unite! unsupervised metrics for reinforced summarization models", Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pp.3246~3256 (2019), Association for Computational Linguistics
- See, A., "Get to the point: Summarization with pointer-generator networks", Proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: Long papers), pp.1073~1083 (2017), Association for Computational Linguistics
- Sizov, G., "Extraction-based automatic summarization: Theoretical and empirical investigation of summarization techniques", pp.2010
- Sun, T., "BERTScore is unfair: On social bias in language model-based metrics for text generation", Proceedings of the 2022 conference on empirical methods in natural language processing, pp.3726~3739 (2022), Association for Computational Linguistics
- Tang, L., "Understanding factual errors in summarization : Errors, summarizers, datasets, error detectors", arXiv preprint arXiv:2205.12854, ~ (2022)
- Thorne, J., "FEVER: a large-scale dataset for fact extraction and VERification", Proceedings of the 2018 conference of the north American chapter of the association for computational linguistics: Human language technologies, 1, pp.809~819 (2018), Association for Computational Linguistics
- Vasilyev, O., "Fill in the blanc : Human-free quality estimation of document summaries", arXiv preprint arXiv:2002.09836, ~ (2020)
- Vaswani, A., "Attention is all you need", Advances in neural information processing systems, 30 (2017)
- Wang, A., "Asking and answering questions to evaluate the factual consistency of summaries", Proceedings of the 58th annual meeting of the association for computational linguistics, pp.5008~5020 (2020), Association for Computational Linguistics
- Wang, A., "GLUE: A multi-task benchmark and analysis platform for natural language understanding", Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP, pp.353~355 (2018), Association for Computational Linguistics
- Yu, W., "A survey of knowledge-enhanced text generation", ACM Computing Surveys, 54(11s), pp.1~38 (2022)
- Yuan, W., "Advances in neural information processing systems (Vol. 34)", pp.27263~27277 (2021), Curran Associates, Inc
- Zaheer, M., "Big bird: Transformers for longer sequences", Advances in Neural Information Processing Systems, 33 (2020)
- Zhang, J., "Pegasus: Pre-training with extracted gap-sentences for abstractive summarization", Proceedings of the 37th international conference on machine learning (2020)
- Zhang, T., "Bertscore: Evaluating text generation with bert", International conference on learning representations (2020)
- Zhu, C., "Machine reading comprehension", pp.185~207 (2021), Elsevier
- Zhu, C., "Enhancing factual consistency of abstractive summarization", Proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: Human language technologies, pp.718~733 (2021), Association for Computational Linguistics
- 김탁영, "한국어 문서요약 및 음성합성 통합 프레임워크 구축", 대한산업공학회지, 48(1), pp.80~90 (2022), 대한산업공학회[DOI]
- 박은환, "Summary-to-document 를 이용한 텍스트 생성 요약", 한국정보과학회 학술발표논문집, pp.308~310 (2021)
- 최경호, "Copy mechanism과 input feeding을 이용한 end-to-end 한국어 문서요약", 한국어정보학회 학술대회, pp.56~61 (2016)
- Katherine Lee, "Hallucinations in neural machine translation", NIPS 2018 Interpretability and Robustness for Audio, Speech and Language Workshop (2018)
