번역자동평가에서 풀리지 않은 과제
Application of Automatic Evaluation to Human Translation
0
조회
0
다운로드
12
인용
0
참고문헌
키워드
초록
This paper deals with two questions. The first is whether BLEU and METEOR, which were developed to evaluate machine translation, can also be used for the evaluation of human translations. The second is, how can these systems be adapted to evaluate human translations in a more variable way? These questions can be subdivided into the following questions: (1) What is the more variable evaluation system, BLEU or METEOR? (2) Is the present precision-recall ratio appropriate? (3) Between the grades and ranks of automatic evaluation, which correlates better with human evaluation? Five translator trainees majored respectively in Arabic, German, English, Japanese, and Spanish (a total of 25 students), translated four texts into Korean (a total of 100 texts). The translations were evaluated by two professional translators in each language and their evaluation results were compared with the outcome of the automatic evaluation. The results showed that the METEOR, recall and ranks correlated with the human ratings better than the BLEU, precision and scores. This and other findings from this experiment suggest that with the minimum number of ca. 12 translations, METEOR can be used at least when determining the order of student performance.
Abstract
This paper deals with two questions. The first is whether BLEU and METEOR, which were developed to evaluate machine translation, can also be used for the evaluation of human translations. The second is, how can these systems be adapted to evaluate human translations in a more variable way? These questions can be subdivided into the following questions: (1) What is the more variable evaluation system, BLEU or METEOR? (2) Is the present precision-recall ratio appropriate? (3) Between the grades and ranks of automatic evaluation, which correlates better with human evaluation? Five translator trainees majored respectively in Arabic, German, English, Japanese, and Spanish (a total of 25 students), translated four texts into Korean (a total of 100 texts). The translations were evaluated by two professional translators in each language and their evaluation results were compared with the outcome of the automatic evaluation. The results showed that the METEOR, recall and ranks correlated with the human ratings better than the BLEU, precision and scores. This and other findings from this experiment suggest that with the minimum number of ca. 12 translations, METEOR can be used at least when determining the order of student performance.
참고문헌 (21)
- 김유섭, "기계 번역에서의 평가", 한국외대 통번역연구소 제18회 국제학술대회 <디지털 시대의 통번역> 발표집, pp.337~357 (2017)
- 김진아, "번역 품질평가에 관한 小考", 외국문학연구, (11), pp.4~124 (2002), 외국문학연구소
- 박혜주, "문학번역 평가 시스템 연구" (2007), 한국문학번역원
- 이영훈, "프랑스 명작소설 한국어 번역 연구를 위한 번역평가 시스템 개발", 통역과 번역, 12(2), pp.149~180 (2010), 한국통역번역학회
- 조성원, "번역평가 기준으로서의 ‘충실성’과 ‘가독성’에 대하여", 안과밖:영미문학연구, (23), pp.96~119 (2007), 영미문학연구회
- 한국외대 번역평가인증 연구팀, "번역인증제도 (실무편)", 한국외대 통번역연구소 학술대회 <언어, 통번역의 평가 및 인증> 발표집, pp.23~33 (2016)
- Banerjee, Satanjeev, "METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments", Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and, 2005, 65~72
- Chung, Hye-Yeon, "Automatische Evaluation der Humanübersetzung: BLEU vs. METEOR", Lebende Sprachen, 65(1) (2020)
- Denkowski, Michael, "Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems", Proceedings of the 6th Workshop on Statistical Machine Translation, pp.85~91 (2011)
- Han, Aaron L, "LEPOR: A Robust Evaluation Metric for Machine Translation with Augmented Factors", Proceedings of COLING 2012: Posters, pp.441~450 (2012)
- House, Juliane, "Translation Quality Assessment: A Model Revisited" (1997), Tübingen
- Jakobsen, Arnt Lykke, "Methods and Strategies of Process research: Integrative Approaches in Translation Studies", pp.37~55 (2011), John Benjamins
- Kunilovskaya, Maria, "How far do we agree on the quality of translation?", English Studies at NBU, 1(1), pp.18~31 (2015)
- Lai, Tzu-Yun, "Reliability and Validity of a Scale-based Assessment for Translation Tests", Meta, 56(3), pp.713~722 (2011)
- "The Significance of Recall in Automatic Metrics for MT Evaluation" (2004)[URL]
- PACTE, "Results of the Validation of the PACTE Translation Competence Model. Decision Making", AILA World Congress 2008. Multilingualism, Challenges & Opportunities, pp.1~5 (2008)
- Papineni, Kishore, "BLEU: a Method for Automatic Evaluation of Machine Translation", Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL),, pp.311~318 (2002)
- "The truth of the F-measure" (2007)[URL]
- Waddington, Christopher, "Should translations be assessed holistically or through error analysis? Hermesi", Journal of Linguistics, 26, pp.15~37 (2001)
- Waddington, Christopher, "Different Methods of Evaluating Student Translations: The Question of Validity", Meta, 46(2), pp.311~325 (2001)
- Williams, Malcolm, "The Application of Argumentation Theory to Translation Quality Assessment", Meta, 46(2), pp.326~344 (2001)