인간 번역평가에서 재현도(recall)의 중요성
Significance of Recall in Automatic Metrics for HT Evaluation
0
조회
0
다운로드
2
인용
0
참고문헌
키워드
초록
In the automatic evaluation of translations, precision and recall are two indices that show how precisely (precision) and how much (recall) the system is able to recognize the well-translated portion in a translation. It would be ideal if two indices could be equally weighted in the evaluation system, since both accuracy and completeness are important criteria in evaluation of human translations (HT). This is, however, not easy, as both indices are negatively correlated. Papineni et al. (2002), for example, opted for precision, while Lavie et al. (2005) used both indices, giving recall nine times more weight than precision. The aim of this work is to examine which of the two indices correlates better with evaluation of professional evaluators and how much weight should be given each to precision and to recall. For this purpose, 459 translated texts were rated with precision, recall, F1 (harmonic mean of precision and recall) and Fmean (nine times higher weight on recall) as well as by professional evaluators. The results show that recall correlates better with human evaluation than precision in almost all cases, but not Fmean than F1, which were equivalent in all but one case. They indicate that recall is indeed a more important metric, but the weight as high as nine on recall is not ideal for HT evaluation.
Abstract
In the automatic evaluation of translations, precision and recall are two indices that show how precisely (precision) and how much (recall) the system is able to recognize the well-translated portion in a translation. It would be ideal if two indices could be equally weighted in the evaluation system, since both accuracy and completeness are important criteria in evaluation of human translations (HT). This is, however, not easy, as both indices are negatively correlated. Papineni et al. (2002), for example, opted for precision, while Lavie et al. (2005) used both indices, giving recall nine times more weight than precision. The aim of this work is to examine which of the two indices correlates better with evaluation of professional evaluators and how much weight should be given each to precision and to recall. For this purpose, 459 translated texts were rated with precision, recall, F1 (harmonic mean of precision and recall) and Fmean (nine times higher weight on recall) as well as by professional evaluators. The results show that recall correlates better with human evaluation than precision in almost all cases, but not Fmean than F1, which were equivalent in all but one case. They indicate that recall is indeed a more important metric, but the weight as high as nine on recall is not ideal for HT evaluation.
참고문헌 (19)
- 권철민, "파이썬 머신러닝 완벽 가이드" (2020), 위키북스
- 정혜연, "번역자동평가에서 풀리지 않은 과제", 번역학연구, 21(1), pp.9~29 (2020), 한국번역학회[DOI]
- 문학번역 평가 시스템 연구, "박혜주", 한국문학번역원, ~ (2007)
- 정혜연, "인간번역 자동평가에서 정답자와 평가자가 다르다면", 독일언어문학, (93), pp.75~95 (2021), 한국독일언어문학회[DOI]
- 정혜연, "임베딩을 활용한 인간번역의 자동평가 - 기계가 의미를 평가할 수 있을까", 통번역학연구, 25(3), pp.141~162 (2021), 통번역연구소[DOI]
- 한국외대 번역평가인증 연구팀, "번역인증제도 (실무편)", 한국외대 통번역연구소 학술대회 <언어, 통번역의 평가 및 인증> 발표집, pp.23~33 (2016)
- Banerjee, Satanjeev, "METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments", Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and, 2005, 65~72
- Buckland, Michael, "The Relationship between Recall and Precision", Journal of the American Society for Information Science, 45(1), pp.12~19 (1994)
- Chung, Hye-Yeon, "Automatische Evaluation der Humanübersetzung: BLEU vs. METEOR", Lebende Sprachen, 65(1), pp.181~205 (2020)
- Han, Lifeng, "Machine Translation Evaluation Resources and Methods: A Survey", IPRC-2018 (Ireland Postgraduate Research Conference) (2018)
- Kunilovskaya, Maria, "How Far Do We Agree on the Quality of Translation?", English Studies at NBU, 1(1), pp.18~31 (2015)
- Lai, Tzu-Yun, "Reliability and Validity of a Scale-based Assessment for Translation Tests", Meta, 56(3), pp.713~722 (2011)
- "The Significance of Recall in Automatic Metrics for MT Evaluation" (2004)[URL]
- Papineni, Kishore, "BLEU: A Method for Automatic Evaluation of Machine Translation", Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pp.311~318 (2002)
- "The Truth of the F-measure" (2007)[URL]
- Waddington, Christopher, "Should Translations Be Assessed Holistically or through Error Analysis?", HERMES Journal of Language and Communication in Business, 26, pp.15~37 (2001)
- Waddington, Christopher, "Different Methods of Evaluating Student Translations: The Question of Validity", Meta, 46(2), pp.311~325 (2001)
- van Rijsbergen, Cornelius, "Information Retrieval" (1979), Butterworth
- Zhang, Tianyi, "BERTScore: Evaluating Text Generation with BERT", Conference Paper at ICLR 2020, pp.1~14 (2020)