임베딩을 활용한 인간번역의 자동평가 - 기계가 의미를 평가할 수 있을까
Automatic Evaluation of Human Translation using Word and Sentence Embedding: Can Machines Evaluate Meaning?
0
조회
0
다운로드
8
인용
0
참고문헌
키워드
초록
This paper begins with the question. Is it possible that a machine can understand and evaluate the meaning of a text? Word2Vec is a neural network method of computing vector representations of words, which means the tool processes a text corpus and produces numeric vectors on the meanings of the words. This paper seeks to examine whether the automatic evaluation metrics, BLEUmodif and METEORmodif, enhanced by word2vec technology, are able to capture the meaning of translated texts better than their standard versions. To this end, first, five literary texts written in English were translated into Korean by 119 students. Then, these translations were evaluated by two professional translators/teachers and by five different automatic evaluation programs. These are standard versions of BLEU and METEOR as well as BLEUmodif and METEORmodif with the word2vec technology and finally Sent2Vec that computes sentence vector representations. For the sake of convenience, the last three systems are referred to as “embedding versions” in this study. The analysis shows that overall, the standard version of METEOR performed the best (grades: 0.838, ranking: 0.727), but at the level of individual texts, the embedding versions of BLEU and METEOR showed a higher correlation with human evaluation (four texts) than their standard versions (one text).
Abstract
This paper begins with the question. Is it possible that a machine can understand and evaluate the meaning of a text? Word2Vec is a neural network method of computing vector representations of words, which means the tool processes a text corpus and produces numeric vectors on the meanings of the words. This paper seeks to examine whether the automatic evaluation metrics, BLEUmodif and METEORmodif, enhanced by word2vec technology, are able to capture the meaning of translated texts better than their standard versions. To this end, first, five literary texts written in English were translated into Korean by 119 students. Then, these translations were evaluated by two professional translators/teachers and by five different automatic evaluation programs. These are standard versions of BLEU and METEOR as well as BLEUmodif and METEORmodif with the word2vec technology and finally Sent2Vec that computes sentence vector representations. For the sake of convenience, the last three systems are referred to as “embedding versions” in this study. The analysis shows that overall, the standard version of METEOR performed the best (grades: 0.838, ranking: 0.727), but at the level of individual texts, the embedding versions of BLEU and METEOR showed a higher correlation with human evaluation (four texts) than their standard versions (one text).
참고문헌 (15)
- 정혜연, "번역자동평가에서 풀리지 않은 과제", 번역학연구, 21(1), pp.9~29 (2020), 한국번역학회[DOI]
- 정혜연, "번역의 자동평가: 기계번역 평가를 인간번역 평가에 적용해보기", 통번역학연구, 22(4), pp.265~287 (2018), 통번역연구소[DOI]
- Banerjee, S., "METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments", Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and, 2005, 65~72
- Chen, B., "Representation-based Translation Evaluation Metrics", Proceedings of the 7th International Joint Conference on Natural Language Processing, pp.150~155 (2015)
- Chung, H.-Y., "Automatische Evaluation der Humanübersetzung: BLEU vs. METEOR", Lebende Sprachen, 65(1), pp.181~205 (2020)
- Denkowski, M., "Meteor Universal : Language Specific Translation Evaluation for Any Target Language", Proceedings of the Ninth Workshop on Statistical Machine Translation, pp.376~380 (2014)
- "Machine Translation Evaluation Resources and Methods: A Survey" (2017)[URL]
- Lane, H., "파이썬으로 배우는 자연어 처리 인 액션" (2018), 제이펍
- Mikolov, T., "Distributed Representations of Words and Phrases and their Compositionality", ~ (2013)
- Mikolov, T., "Distributed Representations of Sentences and Documents", Proceedings of the 31 st International Conference on Machine Learning, 32 (2014), JMLR: W&CP
- Papineni, K., "BLEU: a Method for Automatic Evaluation of Machine Translation", Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pp.311~318 (2002)
- Servan, C., "Word2Vec vs DBnary: Augmenting METEOR using Vector Representations or Lexical Resources?", ~ (2016)
- Vela, M., "Predicting Translation Adequacy with Document Embeddings", Proceedings of the 10th Workshop on Statistical Machine Translation, pp.402~410 (2015)
- Wang, H., "Modification of Machine Translation Evaluation Metrics by Using word Embeddings", Proceeding of the Sixth Workshop on Hybrid Approaches to Translation, pp.33~41 (2016)
- "Google’s Neural Machine Translation System:Bridging the Gap between Human and Machine Translation" (2016)[URL]