NMT의 평가항목별 자동평가를 위한 영한 평가세트 연구
A Study on English-to-Korean Test Suites for NMT Automatic Evaluation by Linguistic Assessment Items
0
조회
0
다운로드
3
인용
0
참고문헌
키워드
초록
This paper describes an approach to automatically evaluate Neural Machine Translation(NMT) systems by linguistic assessment items. While the previous automatic evaluation approaches cannot identify the strengths and weaknesses of NMT systems for each linguistic assessment item, our automatic evaluation approach can intuitively determine both strengths and weaknesses of each linguistic assessment item. The automatic evaluation by linguistic assessment items of NMT systems is evaluated based on whether the answer of translation exists in the machine translation results, after building the test suites of the source text, the expressions in the source text, and the translated word. As applying the automatic evaluation approach by linguistic assessment items to NMT systems of Papago by Naver and Google Translate by Google, we figured out the strengths and weaknesses of each system. The biggest weakness of Papago English-to-Korean machine translation system is Cohesion(40.00%). The most serious weak points of Google English-to-Korean translation system are the translation of Relative pronoun(35.00%), Spoken expression(40.00%), Structural Ambiguity (40.00%), and Cohesion(40.00%). The main purpose of automatic evaluation by the linguistic assessment items is to find various weaknesses of the machine translation systems, semi-automatically collect and build a targeted corpus based on the weaknesses, and improve the performance of the machine translation systems incrementally by retraining. Although this paper has the advantage of automatically recognizing the strengths and weaknesses by linguistic assessment items, the simplified automatic evaluation approach, a measurement based on the matching of translated word and machine translation, that this paper suggests should be improved. In this respect, the improvement directions of this paper in the future are 1) enlarging the linguistic assessment items to other language pairs other than English-to-Korean, 2) semi-automatically collecting the source text which is targeted for evaluation, 3) extending the research to machine interpreting with speech data, 4) including the assessment items that human translator considers.
Abstract
This paper describes an approach to automatically evaluate Neural Machine Translation(NMT) systems by linguistic assessment items. While the previous automatic evaluation approaches cannot identify the strengths and weaknesses of NMT systems for each linguistic assessment item, our automatic evaluation approach can intuitively determine both strengths and weaknesses of each linguistic assessment item. The automatic evaluation by linguistic assessment items of NMT systems is evaluated based on whether the answer of translation exists in the machine translation results, after building the test suites of the source text, the expressions in the source text, and the translated word. As applying the automatic evaluation approach by linguistic assessment items to NMT systems of Papago by Naver and Google Translate by Google, we figured out the strengths and weaknesses of each system. The biggest weakness of Papago English-to-Korean machine translation system is Cohesion(40.00%). The most serious weak points of Google English-to-Korean translation system are the translation of Relative pronoun(35.00%), Spoken expression(40.00%), Structural Ambiguity (40.00%), and Cohesion(40.00%). The main purpose of automatic evaluation by the linguistic assessment items is to find various weaknesses of the machine translation systems, semi-automatically collect and build a targeted corpus based on the weaknesses, and improve the performance of the machine translation systems incrementally by retraining. Although this paper has the advantage of automatically recognizing the strengths and weaknesses by linguistic assessment items, the simplified automatic evaluation approach, a measurement based on the matching of translated word and machine translation, that this paper suggests should be improved. In this respect, the improvement directions of this paper in the future are 1) enlarging the linguistic assessment items to other language pairs other than English-to-Korean, 2) semi-automatically collecting the source text which is targeted for evaluation, 3) extending the research to machine interpreting with speech data, 4) including the assessment items that human translator considers.
참고문헌 (23)
- 강수정, "전문번역사들의 NMT에 대한 인식과 수용에 관한 연구 —심층인터뷰를 중심으로", 번역학연구, 21(3), pp.9~35 (2020), 한국번역학회[DOI]
- 정혜연, "번역자동평가에서 풀리지 않은 과제", 번역학연구, 21(1), pp.9~29 (2020), 한국번역학회[DOI]
- 서보현, "기계번역 결과물의 오류유형 고찰", 번역학연구, 19(1), pp.99~117 (2018), 한국번역학회[DOI]
- 정혜연, "번역의 자동평가: 기계번역 평가를 인간번역 평가에 적용해보기", 통번역학연구, 22(4), pp.265~287 (2018), 통번역연구소[DOI]
- 최동익, "무생물 주어 구문에 대한 영한 기계 번역 오류분석", 언어학 연구, (29), pp.279~299 (2013), 한국중원언어학회[DOI]
- Bentivogli, L, "Neural versus phrase-based machine translation quality: a case study’", Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp.257~267 (2016)
- Bojar O, "Analysing Error Types in English-Czech Machine Translation", The Prague Bulletin of Mathematical Linguistics, pp.63~76 (2011)
- Brussel, Laura Van, "A fine-grained error analysis of NMT, PBMT and RBMT output for English-to-Dutch", Proceedings of the Eleventh International Conference on Language Resources and Evaluation, pp.3799~3804 (2018)
- Costa, Angela, "A Linguistically Motivated Taxonomy for Machine Translation Error Analysis", Machine Translation, 29(2), pp.127~161 (2015)
- Daems, Joke, "On the origin of errors: a fine-grained analysis of MT and PE errors and their relationship", Proceedings of the Ninth International Conference on Language Resources and Evaluation, pp.62~66 (2014)
- Doyon, J, "The DARPA MT evaluation methodology: Past and present", Proceedings of the Association for Machine Translation in the Americas, pp.1~4 (1998)
- Fishel, Mark, "Terra: a Collection of Translation Error-Annotated Corpora’", Proceedings of the Eighth International Conference on Language Resources and Evaluation, pp.7~14 (2012)
- Guillou, L, "PROTEST: A Test Suite for Evaluating Pronouns in Machine Translation", Proceedings of the Tenth International Conference on Language Resources and Evaluation, pp.636~643 (2016)
- Isabelle, P, "A Challenge Set Approach to Evaluating Machine Translation", Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp.2486~2496 (2017)
- Koehn, Philipp, "Six Challenges for Neural Machine Translation", Proceedings of the First Workshop on Neural Machine Translation, pp.28~39 (2017)
- Lommel, A, "Using a New Analytic Measure for the Annotation and Analysis of MT Errors on Real Data", EAMT-2014, pp.165~172 (2014)
- Macketanz, Vivien, "Fine-grained evaluation of German-English Machine Translation based on a Test Suite", Proceedings of the Third Conference on Machine Translation (WMT), 2 (Shared Task Papers), pp.578~587 (2018)
- Papineni, Kishore, "BLEU: a method for automatic evaluation of Machine Translation", Proceedings of the 40th Annual Meeting on Association for Computational Linguistics’, Association for Computational Linguistics, Stroudsburg, PA, USA, ACL ’02, pp.311~318 (2002)
- Popović, Maja, "Towards automatic error analysis of machine translation output’", Computational Linguistics, 37(4), pp.657~688 (2011)
- Snover, Matthew, "A Study of Translation Edit Rate with Targeted Human Annotation", Proceedings of Association for Machine Translation in the Americas, pp.223~231 (2006)
- Vilar D, "Error Analysis of Machine Translation Output", International Conference on Language Resources and Evaluation, pp.697~702 (2006)
- Wisniewski, Guillaume, "A Corpus of Machine Translation Errors Extracted from Translation Students Exercises", Proceedings of the Ninth International Conference on Language Resources and Evaluation, pp.3585~3588 (2014)
- "Google’s neural machine translation system: Bridging the gap between human and machine translation" (2016)[URL]