본문 바로가기
KERD
논문KCI 등재

인공신경망 기반 맞춤형 기계번역엔진의 성능 평가: 법률 및 특허 한영번역 결과물 평가 사례를 중심으로

A Case Study of the Evaluation of Legal and Patent Korean-English Machine Translations by a Domain-Specific NMT Engine

이지은|최효은
한국번역학회 (The Korean Association for Translation Studies)|2023.03|번역학연구 (The Journal of Translation Studies)|제24권 제1호|pp. 9-37|DOI 10.15749/jts.2023.24.1.001

1

조회

0

다운로드

6

인용

0

참고문헌

키워드

MT evaluationNMT법률기계번역특허기계번역자동번역기계번역 평가legal MTpatent MTautomatic translation
주제분류통역번역학

초록

This paper addresses the quality of Korean-English legal and patent translation outputs by a commercial neural machine translation engine customized for legal and patent translations. The current research is based on both automatic and human evaluations of Otran’s English translations of Korean statutes and Korean titles of invention and abstracts extracted from patent gazettes. In automatic evaluation, both BLEU and METEOR scores revealed that legal translation outperformed patent translation. Human evaluation results confirmed the automatic evaluation results, showing Otran’s legal translation receiving better evaluation than its patent translation. According to the error comments provided by evaluators, terminology and other errors, mostly stylistic issues, were the most prevalent error types in the legal translation, while terminology and syntax errors were the most frequent in the patent translation. In the legal translation, accuracy and fluency errors were far scarcer than in the patent translation. The results suggest that the domain-specific NMT engine needs improvement in handling terminology in both legal and patent translation, and its legal translation output proved to be good enough for gisting. The findings from this case study cannot be generalized and thus call for further research.

Abstract

This paper addresses the quality of Korean-English legal and patent translation outputs by a commercial neural machine translation engine customized for legal and patent translations. The current research is based on both automatic and human evaluations of Otran’s English translations of Korean statutes and Korean titles of invention and abstracts extracted from patent gazettes. In automatic evaluation, both BLEU and METEOR scores revealed that legal translation outperformed patent translation. Human evaluation results confirmed the automatic evaluation results, showing Otran’s legal translation receiving better evaluation than its patent translation. According to the error comments provided by evaluators, terminology and other errors, mostly stylistic issues, were the most prevalent error types in the legal translation, while terminology and syntax errors were the most frequent in the patent translation. In the legal translation, accuracy and fluency errors were far scarcer than in the patent translation. The results suggest that the domain-specific NMT engine needs improvement in handling terminology in both legal and patent translation, and its legal translation output proved to be good enough for gisting. The findings from this case study cannot be generalized and thus call for further research.

참고문헌 (50)

  1. 강병규, "신경망 기계번역의 작동 원리와 번역의 정확률 - 중한 번역을 실례로", 중어중문학, (73), pp.253~295 (2018), 한국중어중문학회[DOI]
  2. 기유미, "한중 기계번역 오류의 문형별 비교분석-네이버 파파고 번역기와 구글 번역기를 중심으로-", 중국연구, 74, pp.3~32 (2018), 중국연구소[DOI]
  3. 정혜연, "번역자동평가에서 풀리지 않은 과제", 번역학연구, 21(1), pp.9~29 (2020), 한국번역학회[DOI]
  4. 김순미, "번역학계와 언어서비스업체(LSP)간 산학협력연구: ‘포스트에디팅 생산성’과 ‘기계번역 엔진 성능 비교’", 번역학연구, 20(1), pp.41~76 (2019), 한국번역학회[DOI]
  5. 김한균, "형사절차상 인공지능기반 통·번역 애플리케이션 활용과 피의자 권리보장", 형사소송 이론과 실무, 13(3), pp.147~174 (2021), 한국형사소송법학회[DOI]
  6. 김혜림, "중한 법령 기계번역 포스트에디팅 교육을 위한 예비 연구", 번역학연구, 22(3), pp.65~99 (2021), 한국번역학회[DOI]
  7. 박옥수, "한영 기계 번역에서 ST의 유형적 특징에 따른 번역 오류 분석", 동아인문학, (41), pp.155~183 (2017), 동아인문학회
  8. 서보현, "기계번역 결과물의 오류유형 고찰", 번역학연구, 19(1), pp.99~117 (2018), 한국번역학회[DOI]
  9. 이상빈, "기계번역에 관한 KCI 연구논문 리뷰: 인문학 저널 논문(2011∼2020년 초)의 논의내용과 연구방법을 중심으로", 통역과 번역, 22(2), pp.75~104 (2020), 한국통역번역학회[DOI]
  10. 이준호, "신경망기계번역의 객관적 평가를 위한 예비연구: 자동평가와 수동평가의 균형점", 통번역학연구, 23(3), pp.171~202 (2019), 통번역연구소[DOI]
  11. 이준호, "법률 특화 번역엔진 성능 평가 - 한영 계약서 번역을 중심으로 -", T&I REVIEW, 12(1), pp.169~192 (2022), 통역번역연구소[DOI]
  12. 이준호, "풀 포스트에디팅에 대한 고찰—풀 포스트에디팅 생산성에 영향을 주는 요소를 중심으로", 번역학연구, 23(5), pp.119~146 (2022), 한국번역학회[DOI]
  13. 이정민, "정보기술 발달에 따른 사법통역의 현재와 미래", 4차산업혁명 법과 정책, 2, pp.171~202 (2020)
  14. 이지은, "인공신경망 특허 기계번역 성능에 관한 연구 - Patent Translate와 WIPO Translate 한영 번역 결과물의 누락과 통사 오류 분석을 중심으로 -", T&I REVIEW, 12(2), pp.105~130 (2022), 통역번역연구소[DOI]
  15. 이지은, "기계번역에서 이중주어 구문의 한영 번역 양상- 구글 번역과 네이버 파파고의 법조문 번역 사례 비교 -", T&I REVIEW, 12(1), pp.211~242 (2022), 통역번역연구소[DOI]
  16. 최문선, "국내 번역학 기계번역 연구 동향:내용 분석과 키워드 분석을 중심으로", 언어학연구, 24(1), pp.275~297 (2019), 한국언어연구학회[DOI]
  17. 최효은, "특허 기계번역 결과물의 평가 - KIPRIS의 무료 한영 기계번역을 중심으로", 통역과 번역, 19(1), pp.139~178 (2017), 한국통역번역학회[DOI]
  18. 한현희, "한노 기계번역의 오류 유형화 및 품질 개선을 위한 프리에디팅(pre-editing) 규칙 제안", 통번역학연구, 23(3), pp.291~327 (2019), 통번역연구소[DOI]
  19. 한현희, "한-노 기계 번역, 어디까지 왔나?:Google과 Papago 번역 성능 비교를 기반으로", 노어노문학, 32(3), pp.63~94 (2020), 한국노어노문학회
  20. Brkić, "Automatic and Human Evaluation on English-Croatian Legislative Test Set", Computer Science, 7816, pp.311~378 (2013)
  21. Carmo, Félix do, "Debunking No Language Left Behind, Human Parity and Other Machine Translation Myths", University of Surrey Centre for Translation Studies LITHME Seminar (2022)
  22. Castilho, Sheila, "Is Neural Machine Translation the New State of the Art?", The Prague Bulletin of Mathematical Linguistics, 108, pp.109~120 (2017)
  23. Chatzikoumi, Eirini, "How to Evaluate Machine Translation: A Review of Automated and Human Metrics", Natural Language Engineering, 26, pp.137~161 (2020)
  24. Defauw, Arne, "Developing a Neural Machine Translation System for Irish", Proceedings of the 2nd Workshop on Technologies for MT of Low Resource Languages, pp.32~38 (2019)
  25. Doherty, Stephen, "Human Issues in Translation Technology", pp.131~148 (2017), Routledge
  26. Giordano, Stella Szantova, "It’s All Greek to Me: Are Attorneys Who Engage in or Procure Legal Translation for Their Clients at Risk of Committing an Ethical Violation?", Quinnipiac Law Review, 31(2), pp.447~487 (2013)
  27. "Achieving Human Parity on Automatic Chinese to English News Translation" (2018)[URL]
  28. Hutchins, W. John, "An Introduction to Machine Translation" (1992), Academic Press
  29. Killman, Jeffrey, "Vocabulary Accuracy of Statistical Machine Translation in the Legal Context", Third Workshop on Post Editing Technology and Practice (Proceedings of the 11th Conference of the Association for Machine Translation in the Americas), pp.85~98 (2014)
  30. Kinoshita, Satoshi, "Comparison of SMT and NMT Trained with Large Patent Corpora: Japio at WAT2017", Proceedings of the 4th Workshop on Asian Translation, pp.140~145 (2017)
  31. Kit, Chunyu, "Comparative Evaluation of Online Machine Translation Systems with Legal Texts", Law Library Journal, 100(2), pp.299~321 (2008)
  32. Koehn, Phillip, "Six Challenges for Neural Machine Translation", Proceedings of the First Workshop on Neural Machine Translation Association for Computational Linguistics, pp.28~39 (2017)
  33. Moorkens, Joss, "Under Pressure : Translation in Times of Austerity", Perspectives, 25(3), pp.464~477 (2017)
  34. Nurminen, Mary, "Raw Machine Translation Use by Patent Professionals : A Case of Distributed Cognition", Translation, Cognition & Behavior, 3(1), pp.100~121 (2020)
  35. Olohan, Maeve, "Scientific and Technical Translation" (2015), Routledge
  36. Poliquen, Bruno, "Full-text Patent Translation at WIPO: Scalability, Quality and Usability", Proceedings of the 6th Workshop on Patent and Scientific Literature Translation (2015)
  37. Poliquen, Bruno, "WIPO Translate: Patent Neural Machine Translation Publicly Available in 10 Languages", Patent and Scientific Literature Translation Workshop at MT Summit Conference (2017)
  38. Premoli, Valeria, "MTPE in Patents: A Successful Business Story", Proceedings of MT Summit XVII, 2, pp.36~41 (2019)
  39. "How Automation through Neural Machine Translation Might Change the Skill Sets of Translators, Draft article written as part of the project Language Competence and Work"[URL]
  40. Rivera-Trigueros, Irene, "Machine Translation Systems and Quality Assessment : A Systemic Review", Language Resources & Evaluation, 56, pp.593~619 (2022)
  41. Ragni, Valentina, "What Has Changed with Neural Machine Translation? A Critical Review of Human Factors", Perspectives, 30(1), pp.137~158 (2022)
  42. "Machine vs. Human Translation: When to Use Which for Legal Translation" (2022)[URL]
  43. Rossi, Laura, "Applicability and Application of Machine Translation Quality Metrics in the Patent Field", World Patent Information, 35, pp.115~125 (2013)
  44. Somers, Harold, "The Use of Machine Translation by Law Librarians ― A Reply to Yates", Law Library Journal, 99(3), pp.611~620 (2007)
  45. Smyth, Darren, "Rage against the(Translation)Machine", Journal of Intellectual Property Law & Practice, 10(3), pp.153~154 (2015)
  46. Tsai, Yvonne, "Linguistic Evaluation of Translation Errors in Chinese ― English Machine Translations of Patent Titles", Forum, 15(1), pp.142~156 (2017)
  47. Vieira, Lucas Nunes, "Machine Translation in the News : A Framing Analysis the Written Press", Translation Spaces, 9(1), pp.98~122 (2020)
  48. Vieira, Lucas Nunes, "Understanding the Societal Impacts of Machine Translation : A Critical Review of the Literature on Medical and Legal Use Cases", Information, Communication & Society, 24(11), pp.1515~1531 (2021)
  49. Wiesmann, Eva, "Machine Translation in the Field of Law : A Study of the Translation of Italian Legal Texts into German", Comparative Legilinguistics, 37, pp.117~153 (2019)
  50. Yates, Sarah, "Scaling the Tower of Babel Fish : An Analysis of the Machine Translation of Legal Information", Law Library Journal, 98(3), pp.481~502 (2006)

관련 논문AI추천

A case study on the evaluation of Korean-English legal translations by generic and custom neural machine translation engines이지은, 최효은한국통역번역학회 · 통역과 번역2023 · 제25권 제1호특허 기계번역 결과물의 평가 - KIPRIS의 무료 한영 기계번역을 중심으로이지은, 최효은한국통역번역학회 · 통역과 번역2017 · 제19권 제1호자동화된 기계학습(AutoML)을 활용한 특허 특화 번역엔진의 영한번역 성능 평가이준호, 이청호, 최효은한국번역학회 · 번역학연구2023 · 제24권 제2호법률 특화 번역엔진 성능 평가 - 한영 계약서 번역을 중심으로 -이준호통역번역연구소 · T&I REVIEW2022 · 제12권 제1호인간번역과 기계번역 및 포스트에디팅의 오류 비교: 한영 법률 번역 사례를 중심으로이지은, 최효은한국번역학회 · 번역학연구2024 · 제25권 제1호The quality and productivity of post-edited machine translation in comparison with human translation: A case of Korean-English legal translation이지은, 최효은한국통역번역학회 · 통역과 번역2024 · 제26권 제1호

이 간행물의 다른 논문 (8건)

한국의 제도 법률번역에 대한 고찰: 개념, 범주, 품질지표유정주pp. 39-70학부 교양영어 수업에서의 기계번역 포스트에디팅: 사용자 인식과 오류를 중심으로박혜선, 최진실pp. 71-107학부생 번역자는 기계번역 포스트에디팅 가이드라인을 어떻게 이해하는가? 중국어-한국어 파파고 기계번역을 중심으로김혜림pp. 109-135현장 기반 프로젝트 실행을 통한 라이선스 뮤지컬 번역 교육 연구: <작은 아씨들> 사례를 중심으로강연종, 이지민pp. 137-161영한 문학번역에서 문체 반영의 평가도구로서 그라이스 격률의 유용성: The Old Man and the Sea의 인용동사를 중심으로구상본pp. 163-191
원문 보기