한국어 생성 요약 성능 평가 지표 분석 연구
A Study on Automatic Metrics for Korean Text Abstractive Summarization
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
본 연구는 한국어 생성 요약의 자동 평가 지표를 분석하고 검증하는 것을 목표로 한다. 언어마다 고유한 특성이 다르므로 각 언어에 적합한 평가지표의 필요성에 따라 한국어에 특화된 연구가 요구된다. 현재 한국어를 대상으로 한 생성 요약 및 메타 평가 연구는 다른 언어에 비해 훨씬 부족한상황이다. 따라서 한국어 생성 요약 데이터를 활용하여 평가 기준 및 문서 유형에 따른 신뢰성 있는 자동 평가 지표를 검증함으로써 향후 생성 요약및 자연어 생성 분야의 한국어 모델 연구에 이바지하고자 한다. 요약 모델 평가 시 공신력 있는 지표로 여겨지는 인간 평가(Human Evaluation)는시간과 비용이 많이 소요되므로 자동 평가 지표 연구는 효율성 측면에서도 중요한 의의가 있다. 10가지 한국어 문서와 참조 요약문, 세 가지 모델(T5,KoBART, GPT-3.5 Turbo) 생성 요약문을 대상으로 유창성, 일관성, 관련성 기준으로 인간 평가와 자동 평가 지표의 상관계수를 산출하였다. 평가기준별 상관 분석 결과, T5 요약문에서는 일관성, 관련성에서 각각 0.33, 0.26, KoBART 요약문에서는 유창성, 관련성에서 0.33, 0.40의 상관계수와함께 BERTScore가 제일 높은 상관관계를 보여 한국어 요약문 평가에 효과적인 지표임을 확인하였다. 한편, 대규모 언어모델인 GPT-3.5 Turbo 요약문은환각 가능성 감지를 위해 개발된 평가 지표 HaRiM+가 일관성, 관련성 측면에서 0.23, 0.17의 유의미한 상관관계를 보였다. 또한, 문서 유형별 상관분석 결과, T5 요약문은 보도자료와 회의록에서 BLEU 지표와 높은 상관관계를 나타냈고, KoBART 요약문은 나레이션 문서에서, GPT-3.5 Turbo요약문은 사설 문서에서 BERTScore와 높은 상관관계를 보였다. 이러한 결과는 특정 문서 유형에 적합한 평가 지표를 선택하는 것이 중요하다는점을 강조한다. 이와 같이, 본 연구는 향후 한국어 요약 연구에서 목표 과제의 목적에 따라 적합한 평가 지표를 선택하는 근거로서 활용할 수 있다.
Abstract
This study aims to analyze and validate automatic evaluation metrics for Korean abstractive summarization. The unique linguistic characteristicsof each language require evaluation metrics designed for them, underscoring the importance of research focused on Korean. Research onsummarization and its meta-evaluation is extremely limited, especially for Korean. Therefore, by validating reliable automatic evaluationmetrics using Korean summarization data, this study contributes to future research on Korean models in the fields of natural language generation. Human evaluation, widely regarded as the most reliable metric, is time-consuming and costly. Thus, research into automatic evaluation metricsholds significant importance for efficiency. In this study, summaries from three models—T5, KoBART,and GPT-3.5 Turbo—were evaluatedbased on their fluency, consistency, and relevance using 10 Korean documents and their corresponding reference summaries. Correlationcoefficients were calculated between human evaluations and automatic metrics for fluency, consistency, and relevance. The results showedthat for T5 summaries, the correlation coefficients for consistency and relevance were 0.33 and 0.26, respectively, while for KoBART summaries,the coefficients for fluency and relevance were 0.33 and 0.40, respectively. BERTScore demonstrated the highest correlation, indicating itseffectiveness for Korean summaries. Meanwhile, GPT-3.5 Turbo summaries showed significant correlations of 0.23 and 0.17 in consistencyand relevance using HaRiM+, a metric developed to detect hallucinations in recent work. Additionally, the correlation analysis by documenttype revealed that T5 summaries showed high correlations with the BLEU metric for briefing and meeting minutes, KoBART summaries andGPT-3.5 Turbo summaries both demonstrated high correlations with BERTScore for narrative and editorial documents, respectively. Thesefindings emphasize the importance of selecting evaluation metrics tailored to specific document types. Therefore, this study provides a basisfor selecting appropriate evaluation metrics tailored to the objectives of specific tasks in future Korean summarization research.
참고문헌 (27)
- R. Mihalcea, "Textrank : Bringing order into text", Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.404~411 (2004)
- M. -H. Choi, "Performance Comparison between Language Generation Models for Summarization of Customer Consultation", Proceedings of Symposium of the Korean Institute of communications and Information Sciences (KICS), pp.932~933 (2023)
- J. Lee, "Entity-aware Medical Document Summarization using Large Language Model", The Institute of Electronics and Information Engineers (IEIE)Academic Presentation Paper Collection, pp.2820~2822 (2024)
- J. -W. Lee, "Korean Information Summary System for National R&D Projcet Information Summary", The Korea Institute of information and Communication Engineering (KIICE) Academic Presentation Paper Collection, pp.72~74 (2022)
- E. Kim, "Comparative study of legal document summary method based on pre-trained model", Proceedings of Annual Conference of Korea Information Processing Society (ACK), 28(2), pp.614~617 (2021)
- J. K. Bae, "A Study on the Construction of Financial"Specific Language Model Applicable to the Financial Institutions", Journal of the Korea Industrial Information Systems Research, 29(3), pp.79~87 (2024)
- C. Y. Lin, "ROUGE : A package for automatic evaluation of summaries", Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL), pp.74~81 (2004)
- T. Zhang, "BERTScore : Evaluating Text Generation with BERT", Proceedings of the 8th International Conference on Learning Representations (ICLR) (2020)
- C. van der Lee, "Human evaluation of automatically generated text : Current trends and best practice guidelines", Computer Speech & Language, 67, pp.101151~ (2021)
- M. Gao, "DialSummEval : Revisiting Summarization Evaluation for Dialogues", Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pp.5693~5709 (2022)
- S. Gabriel, "Go Figure! A Meta Evaluation of Factuality in Summarization", Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp.478~487 (2021)
- O. Honovich, "TRUE : Re-evaluating Factual Consistency Evaluation", Proceedings of the DialDoc Workshop at ACL 2022, pp.161~175 (2022)
- "요약문 및 레포트 생성 데이터"[URL]
- "KoBART: Korean Generative Pretrained Transformer"[URL]
- C. Raffel, "Exploring the limits of transfer learning with a unified text-to-text transformer", Journal of Machine Learning Research, 21(140), pp.1~67 (2020)
- "GPT-3.5 Turbo Documentation"[URL]
- W. S. El-Kassas, "Automatic text summarization : A comprehensive survey", Expert Systems with Applications, 165, pp.113679~ (2021)
- K. Papineni, "BLEU : A method for automatic evaluation of machine translation", Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pp.311~318 (2002)
- M. Eyal, "Question Answering as an Automatic Evaluation Metric for News Article Summarization", Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pp.3938~3948 (2019)
- T. Sellam, "BLEURT : Learning robust metrics for text generation", Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pp.7881~7892 (2020)
- S. J. Park, "HaRiM+ : Evaluating Summary Quality with Hallucination Risk", Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp.895~924 (2022)
- J. Devlin, "BERT : Pre-training of deep bidirectional transformers for language understanding", Proceedings of the 2019Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pp.4171~4186 (2019)
- A. Pagnoni, "Understanding factuality in abstractive summarization with FRANK : A benchmark for factuality metrics", Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pp.4812~4829 (2021)
- S. Wang, "Asking and answering questions to evaluate the factual consistency of summaries", Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pp.5008~5020 (2020)
- A. R. Fabbri, "SummEval : Re-evaluating summarization evaluation", Transactions of the Association for Computational Linguistics, 9, pp.391~409 (2021)
- I. Sutskever, "Sequence to Sequence Learning with Neural Networks", Proceedings of the 27th International Conference on Neural Information Processing Systems (NIPS 2014), pp.3104~3112 (2014)