번역의 자동평가: 기계번역 평가를 인간번역 평가에 적용해보기
Automatic Evaluation of Human Translation
0
조회
0
다운로드
32
인용
0
참고문헌
키워드
초록
‘Appropriate’ evaluation of translation quality requires three criteria: validity, reliability and practicality. The human translation evaluation can be considered as ‘valid’, if it is conducted by a professional evaluator, but it leaves out the other two criteria of evaluating human translation. Automatic evaluation, on the other hand, fulfills the second and the third, but misses the first criteria. It would be therefore, desirable, if both methods could complement each other. This paper attempts to verify this possibility. Using an online tool (interactive BLEU score evaluator), 150 pairs of translations were automatically evaluated and compared to the ratings of a human evaluator (professional translator and evaluator). The results showed only a slight inter-rater agreement (Kappa = .054, .109), but made us some implicit suggestions about the improvement of Kappa score. This can be increased, if calculations are based on 1-gram matches(or even morphemes), instead of 1-4 gram matches, and if the reference translations contain a certain quantity of synonyms. With regard to the validity of BLEU, Brevity Penalty should be substituted by a Length Penalty (Han et al., 2012) accordingly to the characteristics of human translation.
Abstract
‘Appropriate’ evaluation of translation quality requires three criteria: validity, reliability and practicality. The human translation evaluation can be considered as ‘valid’, if it is conducted by a professional evaluator, but it leaves out the other two criteria of evaluating human translation. Automatic evaluation, on the other hand, fulfills the second and the third, but misses the first criteria. It would be therefore, desirable, if both methods could complement each other. This paper attempts to verify this possibility. Using an online tool (interactive BLEU score evaluator), 150 pairs of translations were automatically evaluated and compared to the ratings of a human evaluator (professional translator and evaluator). The results showed only a slight inter-rater agreement (Kappa = .054, .109), but made us some implicit suggestions about the improvement of Kappa score. This can be increased, if calculations are based on 1-gram matches(or even morphemes), instead of 1-4 gram matches, and if the reference translations contain a certain quantity of synonyms. With regard to the validity of BLEU, Brevity Penalty should be substituted by a Length Penalty (Han et al., 2012) accordingly to the characteristics of human translation.
참고문헌 (31)
- 김유섭, "기계 번역에서의 평가", 한국외대 통번역연구소 제18회 국제학술대회 <디지털 시대의 통번역> 발표집, pp.337~357 (2017)
- 김진아, "번역 품질평가에 관한 小考", 외국문학연구, (11), pp.4~124 (2002), 외국문학연구소
- 문학번역 평가 시스템 연구, "박혜주", 한국문학번역원, ~ (2007)
- 손지봉, "문학번역 평가기준에 관하여", 통역과 번역, 8(1), pp.67~106 (2006), 한국통역번역학회
- 이영훈, "프랑스 명작소설 한국어 번역 연구를 위한 번역평가 시스템 개발", 통역과 번역, 12(2), pp.149~180 (2010), 한국통역번역학회
- 조성원, "번역평가 기준으로서의 ‘충실성’과 ‘가독성’에 대하여", 안과밖:영미문학연구, (23), pp.96~119 (2007), 영미문학연구회
- 한국외대 번역평가인증 연구팀, "번역인증제도 (실무편)", 한국외대 통번역 연구소 학술대회 <언어, 통번역의 평가 및 인증> 발표집, pp.23~33 (2016)
- Baker, M., "Terminology, LSP and translation: studies in language engineering in Honour of Juan C. Sager", pp.175~186 (1996), John Benjamins
- Banerjee, S., "METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments", Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and, 2005, 65~72
- Brunette, L., "Towards a Terminology for Translation Quality Assessment", The Translator, 6(2), pp.169~182 (2000)
- "Re-evaluating the Role of Bleu in Machine Translation Research" (2006)[URL]
- Denkowski, M., "Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems", Proceedings of the 6th Workshop on Statistical Machine Translation, pp.85~91 (2011)
- Echizen’ya, H., "Automatic Evaluation Metric for Machine Translation that is Independent of Sentence Length", Proceedings of Recent Advances in Natural Language Processing, pp.230~236 (2013)
- Gerzymisch-Arbogast, H., "Equivalence parameter and evaluation", Meta, 46(2), pp.227~242 (2001)
- "Machine Translation Evaluation Resources and Methods: A Survey" (2017)[URL]
- Han, A. L., "LEPOR: A Robust Evaluation Metric for Machine Translation with Augmented Factors", Proceedings of COLING 2012: Posters, pp.441~450 (2012)
- House, J., "Translation Quality Assessment: A Model Revisited" (1997), Gunter Narr
- Landis, J. R., "The Measurement of Observer Agreement for Categorical Data", Biometrics, 33, pp.159~174 (1977)
- Nida, E., "The Theory and Practice of Translation" (1969), the United Bible Societies
- Papineni, K., "BLEU: a Method for Automatic Evaluation of Machine Translation", Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL) (2002)
- Reiss, K., "Möglichkeiten und Grenzen der Übersetzungskritik" (1971), Hueber
- Tan, L., "An Awkward Disparity between BLEU", RIBES Scores and Human Judgements in Machine Translation, 2015, 74~81[DOI]
- Turian, J. P., "Evaluation of Machine Translation and its Evaluation", Proceedings of MT Summit IX (2003), Association for Machine Translation in the Americas
- Vermeer, H., "Grundlegung einer allgemeinen Translationstheorie" (1984), Max Niemeyer Verlag
- Virpioja, S., "LeBLEU: N-gram-based Translation Evaluation Score for Morphologically Complex Languages", Proceedings of the Tenth Workshop on Statistical Machine Translation, pp.411~416 (2015)
- Waddington, C., "Different Methods of Evaluating Student Translations: The Question of Validity", Meta, 46(2), pp.311~325 (2001)
- "Comparison and Adaptation of Automatic Evaluation Metrics for Quality Assessment of Re-Speaking" (2016)[URL]
- "CNL"[URL]
- "CTTIC"[URL]
- "NAATI"[URL]
- "Tilde interactive BLEU score evaluator"[URL]