본문 바로가기
KERD
논문KCI 등재

A case study on the evaluation of Korean-English legal translations by generic and custom neural machine translation engines

이지은|최효은
한국통역번역학회 (Korean Society of Interpretation and Translation Studies)|2023.04|통역과 번역 (Interpretation and Translation)|제25권 제1호|pp. 75-98|DOI https://doi.org/10.20305/it202301075098

1

조회

0

다운로드

5

인용

0

참고문헌

키워드

custom enginegeneric engineKorean-English legal translationmachine translation evaluationstatutes
주제분류통역번역학

초록

This paper addresses the quality of Korean-English legal translation outputs generated by three neural machine translation engines, namely NMT1, NMT2 and NMT3. NMT1 and NMT2 are widely available multilingual generic machine translation engines, whereas NMT3 is a custom machine translation engine for legal translation. In both automatic and human evaluations of the three engines’ English translations of Korean statutes, the custom engine outperformed the other two NMTs. BLEU and METEOR scores revealed that NMT3’s output was more similar to human reference translation. Human evaluation results confirmed NMT3’s strong performance, outperforming the generic engines in accuracy and fluency while underperforming in terminology. However, further analysis revealed that NMT3 was superior in handling terms comprising compound nouns and sentences over 25 words. Although the custom engine’s performance still requires substantial human involvement for quality translation, it showed a potential for the legal domain, demonstrating human parity in short sentences. The growing demand for speedy translation under budget constraints calls for further research on technology-based translation tools and human-computer interaction for legal translation.

Abstract

This paper addresses the quality of Korean-English legal translation outputs generated by three neural machine translation engines, namely NMT1, NMT2 and NMT3. NMT1 and NMT2 are widely available multilingual generic machine translation engines, whereas NMT3 is a custom machine translation engine for legal translation. In both automatic and human evaluations of the three engines’ English translations of Korean statutes, the custom engine outperformed the other two NMTs. BLEU and METEOR scores revealed that NMT3’s output was more similar to human reference translation. Human evaluation results confirmed NMT3’s strong performance, outperforming the generic engines in accuracy and fluency while underperforming in terminology. However, further analysis revealed that NMT3 was superior in handling terms comprising compound nouns and sentences over 25 words. Although the custom engine’s performance still requires substantial human involvement for quality translation, it showed a potential for the legal domain, demonstrating human parity in short sentences. The growing demand for speedy translation under budget constraints calls for further research on technology-based translation tools and human-computer interaction for legal translation.

참고문헌 (43)

  1. Brkić, M., "Automatic and human evaluation on English-Croatian legislative test set", Computer Science, 7816, pp.311~378 (2013)
  2. Castilho, S., "Translation Quality Assessment: From Principles to Practice", pp.9~38 (2018), Springer
  3. Castilho, S., "Reading comprehension of machine translation output: What makes for a better read?", Proceedings of the 21st Annual Conference of the European Association for Machine Translation, pp.79~88 (2018), European Association for Machine Translation
  4. Chatzikoumi, E., "How to evaluate machine translation: A review of automated and human metrics", Natural Language Engineering, 26, pp.137~161 (2020)
  5. Chunyu, K., "Comparative evaluation of online machine translation systems with legal texts", Law Library Journal, 100(2), pp.299~321 (2008)
  6. Costam Â., "A linguistically motivated taxonomy for Machine Translation error analysis", Machine Translation, 29(2), pp.127~161 (2015)
  7. Defauw, A., "Developing a neural machine translation system for Irish", Proceedings of the 2nd Workshop on Technologies for MT of Low Resource Languages, pp.32~38 (2019), European Association for Machine Translation
  8. Doherty, S, "Human Issues in Translation Technology", pp.131~148 (2017), Routledge
  9. Enríquez Raído, V., "Translators as adaptive experts in a flat world: From Globalization 1.0 to Globalization 4.0?", International Journal of Communication, 10, pp.970~988 (2016)
  10. Guo, J., "Is Google Translate adequate for facilitating instrument translation from English to Mandarin", Cin-Computers Informatics Nursing, 34(9), pp.377~383 (2016)
  11. Killman, J., "Vocabulary accuracy of statistical machine translation in the legal context", Third Workshop on Post Editing Technology and Practice, pp.85~98 (2014)
  12. 김순미, "번역학계와 언어서비스업체(LSP)간 산학협력연구: ‘포스트에디팅 생산성’과 ‘기계번역 엔진 성능 비교’", 번역학연구, 20(1), pp.41~76 (2019), 한국번역학회[DOI]
  13. Kit, C., "Routledge Handbook of Translation and Technology", pp.213~236 (2014), Routledge
  14. Klubicka, F., "Quantitative fine-grained human evaluation of machine translation systems : A case study on English to Croatian", Machine Translation, 32, pp.195~215 (2017)
  15. Koehn, P., "Six challenges for neural machine translation", Proceedings of the First Workshop on Neural Machine Translation, pp.28~39 (2017), Association for Computational Linguistics
  16. 구명철, "법률 언어의 가독성 향상을 위한 통사론적인 기초연구 - 법조문에서의 주어생략 현상을 중심으로", 독어학, (38), pp.23~46 (2018), 한국독어학회[DOI]
  17. 곽중철, "포스트에디팅 측정지표를 통한 기계번역 오류 유형화 연구", 통번역학연구, 22(1), pp.1~25 (2018), 통번역연구소[DOI]
  18. Läubli, S., "Has machine translation achieved human parity? A case for document-level evaluation", Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.4791~4796 (2018), Association for Computational Linguistics
  19. "MT evaluation: Human measures and assessment methods" (2013)[URL]
  20. 이준호, "신경망기계번역의 객관적 평가를 위한 예비연구: 자동평가와 수동평가의 균형점", 통번역학연구, 23(3), pp.171~202 (2019), 통번역연구소[DOI]
  21. 이준호, "법률 특화 번역엔진 성능 평가 - 한영 계약서 번역을 중심으로 -", T&I REVIEW, 12(1), pp.169~192 (2022), 통역번역연구소[DOI]
  22. 이상모, "기계번역을 활용한 법령번역의 실제와 과제", T&I REVIEW, 11(1), pp.35~56 (2021), 통역번역연구소[DOI]
  23. 이성화, "영-한 및 한-영 기계번역 품질향상을 위한프리에디팅 기법 제안", 번역학연구, 19(5), pp.121~154 (2018), 한국번역학회[DOI]
  24. Fiorenza Mileto, "Post-editing and legal translation", H2D. Revista de Humanidades Digitais, 1(1) (2019), University of Minho[DOI]
  25. Moorkens, J., "What to expect from neural machine translation : A practical in-class translation evaluation exercise", The Interpreter and Translator Trainer, 12(4), pp.375~387 (2018)
  26. Nunes, V., "Understanding the societal impacts of machine translation: A critical review of the literature on medical and legal use cases", Information. Communication & Society, 24(11), pp.1515~1531 (2020)
  27. 박옥수, "한영 병렬 코퍼스와 기계번역에서 의존 명사 ‘-것’이 포함된 어휘의 번역 방식 연구", 동아인문학, (37), pp.469~492 (2016), 동아인문학회
  28. 박옥수, "한영 기계 번역에서 ST의 유형적 특징에 따른 번역 오류 분석", 동아인문학, (41), pp.155~183 (2017), 동아인문학회
  29. 박옥수, "원천언어의 관점에서 살펴본 기계번역의 오류분석과 수정 원칙: 비문학 텍스트의 통사적 특징에 근거해서", 동아인문학, (44), pp.151~171 (2018), 동아인문학회[DOI]
  30. 박영란, "기계 번역에서의 영어속담 분석", 통번역교육연구, 16(1), pp.131~150 (2018), 한국통번역교육학회[DOI]
  31. "Context in Neural Machine Translation: A Review of Models and Evaluations" (2019)[URL]
  32. Qun, L., "The Routledge Encyclopedia of Translation Technology", pp.105~119 (2014), Routledge
  33. Ragni, V., "What has changed with neural machine translation? A critical review of human factors", Perspectives, 30(1), pp.137~158 (2022)
  34. Rivera-Trigueros, I., "Machine translation systems and quality assessment : A systemic review", Language Resources & Evaluation, 56, pp.593~619 (2022)
  35. "Machine vs. Human translation: When to use which for legal translation" (2022)[URL]
  36. Rossi, L., "Applicability and application of machine translation quality metrics in the patent field", World Patent Information, 35, pp.115~125 (2013)
  37. 서보현, "기계번역 결과물의 오류유형 고찰", 번역학연구, 19(1), pp.99~117 (2018), 한국번역학회[DOI]
  38. Toral, A., "Translation Quality Assessment. From Principles to Practice", pp.263~287 (2018), Springer International Publishing
  39. Toral, A., "Post-Editing effort of a Novel with statistical and neural machine translation", Frontiers in Digital Humanities, 5(9), pp.1~11 (2018)
  40. Toral, A., "Attaining the Unattainable? Reassessing Claims of human parity in neural machine translation", Proceedings of the Third Conference on Machine Translation (WMT18), pp.113~123 (2018), Association for Computational Linguistics
  41. Vigier-Moreno, F. C., "Assessing neural machine translation of court documents : a case study on the translation of a spanish remand order into english", Journal of Language and Law, 78, pp.73~91 (2022)
  42. Wiesmann, E., "Machine translation in the field of law : A study of the translation of Italian legal texts into German", Comparative Legilinguistics, 37, pp.117~153 (2019)
  43. Yates, S., "Scaling the tower of Babel Fish : An analysis of the machine translation of legal information", Law Library Journal, 98(3), pp.481~500 (2006)

관련 논문AI추천

인공신경망 기반 맞춤형 기계번역엔진의 성능 평가: 법률 및 특허 한영번역 결과물 평가 사례를 중심으로이지은, 최효은한국번역학회 · 번역학연구2023 · 제24권 제1호법률 특화 번역엔진 성능 평가 - 한영 계약서 번역을 중심으로 -이준호통역번역연구소 · T&I REVIEW2022 · 제12권 제1호The quality and productivity of post-edited machine translation in comparison with human translation: A case of Korean-English legal translation이지은, 최효은한국통역번역학회 · 통역과 번역2024 · 제26권 제1호인간번역과 기계번역 및 포스트에디팅의 오류 비교: 한영 법률 번역 사례를 중심으로이지은, 최효은한국번역학회 · 번역학연구2024 · 제25권 제1호법령의 MTPE 품질 평가 모델 탐색과 제안: 한국어 법령의 중국어 번역을 중심으로김아영, 오성탁한국통역번역학회 · 통역과 번역2025 · 제27권 제3호신경망기계번역의 객관적 평가를 위한 예비연구: 자동평가와 수동평가의 균형점이준호통번역연구소 · 통번역학연구2019 · 제23권 제3호

이 간행물의 다른 논문 (7건)

소셜미디어 상에서의 통역사에 대한 인식과 통역 콘텐츠 분석: 유튜브(Youtube)의 비정형 데이터를 이용한 텍스트 마이닝 분석과 질적 사례연구권상미pp. 1-26통번역대학원 한영번역 수업에서의 플립러닝 적용사례이상애pp. 27-47기계번역 사용, 기계번역 교육, 번역가 진로에 관한 인식 조사: 학부번역전공자를 대상으로 한 소규모 설문조사를 기반으로이선우, 이상빈pp. 49-73중한 번역 그림책에서 멀티모달리티를 구현한 트랜스크리에이션 전략 고찰이지혜pp. 99-130통역 교육에서의 원문 텍스트 난이도 평가 자동화: 코메트릭스(Coh-Metrix)를 이용한 다차원적 텍스트 난이도 평가를 중심으로최문선pp. 131-153
원문 보기