신경망기계번역의 객관적 평가를 위한 예비연구: 자동평가와 수동평가의 균형점
A Pilot Study Toward an Objective Assessment of Neural Machine Translation; Balance between Automatic and Manual Assessment
0
조회
0
다운로드
17
인용
0
참고문헌
키워드
초록
Neural Machine Translation (NMT) is known to produce better translation than Statistic Machine Translation(SMT). It can be natural that scholars and media show more interest in the emerging translation technology. However, it is difficult to define what makes a translation better than others. Furthermore, the claims about the NMT quality and wild guess about the future of translation are sometimes misleading and even worrisome. This means more objective and reliable diagnosis about the current state of NMT is necessary so that potential users and readers of machine translation can make informed decisions. Hence, this research aims to investigate if the media's approach, mainly driven by automatic evaluation, is valid and to review what has been considered as important aspects for translation evaluation in the previous translation studies. As a next step toward a more objective diagnosis, this research tries to compare three Neural Machine Translation (NMT) services available in Korea and assess their translation quality from various perspectives. In total, eight translation experts conducted a series of quality assessment on Korean to English and English and Korean translation. The analysis of the experts input will provide a few meaningful findings that can help build more objective yet prudent views on neural machine translation.
Abstract
Neural Machine Translation (NMT) is known to produce better translation than Statistic Machine Translation(SMT). It can be natural that scholars and media show more interest in the emerging translation technology. However, it is difficult to define what makes a translation better than others. Furthermore, the claims about the NMT quality and wild guess about the future of translation are sometimes misleading and even worrisome. This means more objective and reliable diagnosis about the current state of NMT is necessary so that potential users and readers of machine translation can make informed decisions. Hence, this research aims to investigate if the media's approach, mainly driven by automatic evaluation, is valid and to review what has been considered as important aspects for translation evaluation in the previous translation studies. As a next step toward a more objective diagnosis, this research tries to compare three Neural Machine Translation (NMT) services available in Korea and assess their translation quality from various perspectives. In total, eight translation experts conducted a series of quality assessment on Korean to English and English and Korean translation. The analysis of the experts input will provide a few meaningful findings that can help build more objective yet prudent views on neural machine translation.
참고문헌 (37)
- 곽중철, "포스트에디팅 측정지표를 통한 기계번역 오류 유형화 연구", 통번역학연구, 22(1), pp.1~25 (2018), 통번역연구소[DOI]
- 마승혜, "한영 기계번역 포스트 에디팅에 대한 경험적 고찰: 학부 교육 과정 및 결과를 중심으로", 통번역학연구, 22(1), pp.53~88 (2018), 통번역연구소[DOI]
- 마승혜, "문학작품 기계번역의 한계에 대한 상세 고찰", 통번역학연구, 22(3), pp.65~88 (2018), 통번역연구소[DOI]
- 박옥수, "한영 기계 번역에서 ST의 유형적 특징에 따른 번역 오류 분석", 동아인문학, (41), pp.155~183 (2017), 동아인문학회
- 박재홍, "Word, Subword, Character 수준의 한국어-영어 인공신경망 기계번역 성능 비교", 한국통신학회 학술대회논문집, pp.60~61 (2016)
- 서보현, "기계번역 결과물의 오류유형 고찰", 번역학연구, 19(1), pp.99~117 (2018), 한국번역학회[DOI]
- 송연석, "기계번역 담론에 대한 비판적 고찰", 번역학연구, 19(1), pp.119~145 (2018), 한국번역학회[DOI]
- 이성화, "영-한 및 한-영 기계번역 품질향상을 위한프리에디팅 기법 제안", 번역학연구, 19(5), pp.121~154 (2018), 한국번역학회[DOI]
- 이원기, "개체명 인식과 단어 정렬을 이용한 통계적 기계번역의 성능 향상", 한국정보과학회 학술발표논문집, pp.615~617 (2017)
- 이주리애, "인공신경망 기계 번역의 한일", 일한 번역 품질에 대한 예비연구-품질향상 검토와 교열코드 적용-, 2018, pp.20(1), 43~71[DOI]
- 이준호, "문학번역 적용을 위한 기계번역의 현주소", 통번역학연구, 23(1), pp.143~167 (2019), 통번역연구소[DOI]
- 이창기, "Neural Machine Translation 기반의 영어-일본어 자동번역", 정보과학회지, 33(10), pp.48~52 (2015)
- 최효은, "특허 기계번역 결과물의 평가 - KIPRIS의 무료 한영 기계번역을 중심으로", 통역과 번역, 19(1), pp.139~178 (2017), 한국통역번역학회[DOI]
- 정혜연, "번역의 자동평가: 기계번역 평가를 인간번역 평가에 적용해보기", 통번역학연구, 22(4), pp.265~287 (2018), 통번역연구소[DOI]
- 진용주, "프리에디팅(pre-editing)이 기계번역 품질에 미치는 영향 고찰: 유창성, 충실성, 가독성 비교를 중심으로", 통번역학연구, 22(3), pp.27~64 (2018), 통번역연구소[DOI]
- 한상철, "인공신경망 기계번역 모델의 최적화를 위한 네트워크 구조 설계", 한국정보과학회 학술발표논문집, pp.699~700 (2017)
- Angelelli, C. V., "Testing and assessment in translation and interpreting studies : A call for dialogue between research and practice" (2009), John Benjamins Publishing
- Banerjee, S., "METEOR: An automatic metric for MT evaluation with improved correlation with human judgments", Proceedings of the ACL workshop on intrinsic and extrinsic evaluation measures for machine translation and, 2005, 65~72
- Bentivogli, L., "Neural versus phrase-based machine translation quality: a case study" (2016)
- Bowker, L., "A corpus-based approach to evaluating student translations", The Translator, 6(2), pp.183~210 (2000)
- Calude, A., "Machine translation of various text genres", The New Zealand Linguistic Society Journal, 46, pp.67~94 (2004)
- Coughlin, D., "Correlating automated and human assessments of machine translation quality", Proceedings of MT summit, IX, pp.63~70 (2003)
- Doherty, S., "Issues in human and automatic translation quality assessment", Human Issues in Translation Technology, pp.149~166 (2017)
- Han, L., "Machine translation evaluation resources and methods: A survey" (2016)
- Hönig, H. G., "Positions, power and practice: Functionalist approaches and translation quality assessment", Current Issues in Language & Society, 4(1), pp.6~34 (1997)
- Koehn, P., "Manual and automatic evaluation of machine translation between european languages", Proceedings on the Workshop on Statistical Machine Translation, pp.102~121 (2006)
- Koehn, P., "Statistical machine translation" (2009), Cambridge University Press
- Koglin, A., "An empirical investigation of cognitive effort required to post-edit machine translated metaphors compared to the translation of metaphors", Translation & Interpreting, 7(1), pp.126~ (2015)
- Lin, C. Y., "Automatic evaluation of summaries using n-gram co-occurrence statistics", Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics (2003)
- Translation and cultural adaptation of the survey instruments, "Maxwell, B.", Third international mathematics and science study (TIMSS), 159~169, pp.Third international mathematics and science study (TIMSS) technical report, 1 (1996)
- Noorbehbahani, F., "The automatic assessment of free text answers using a modified BLEU algorithm", Computers & Education, 56(2), pp.337~345 (2011)
- Papineni, K., "BLEU: a method for automatic evaluation of machine translation", Proceedings of the 40th annual meeting on association for computational linguistics, pp.311~318 (2002)
- Plesco, C., "Machine translation as a global enterprise service at ford", Proceedings of the Tenth Conference of the Association for Machine Translation in the Americas (2012)
- Raído, V. E., "Translators as adaptive experts in a flat world: From Globalization 1.0 to Globalization 4.0", International Journal of Communication, 10, pp.970~988 (2016)
- Reiss, K., "Translation criticism-potentials and limitations: Categories and criteria for translation quality assessment" (2014), Routledge
- Schiaffino, R., "Translation quality measurement in practice", Proceedings of the 46th Annual Conference of the American Translation Association (2005)
- "An instructor’s guide to understanding test reliability" (2003)[URL]