버트(BERT)를 활용한 인간번역의 자동평가: 여러 모델의 성능 비교 및 활용 가능성
Automatic Evaluation of Human Translation using BERT
0
조회
0
다운로드
4
인용
0
참고문헌
키워드
초록
The most recent language model in NLP, known as BERT, has numerous advantages over other language models. It ‘understands’ human language in a subword unit and can recognize rare words and out-of-vocabulary words. Furthermore, it considers syntax and context during the text-understanding process. We applied the evaluation metric “BERTscore” using this BERT model to evaluate human translation (HT). 120 translated texts were evaluated with five evaluation metrics: BLEU, METEOR, emBLEU, emMETEOR, as well as BERTscore and the result was compared with professional translators’ evaluation. The comparison examines the validity and reliability of these metrics, particularly the BERTscore, for future application for HT evaluation. BERTscore demonstrated a stable performance, taking first place in scores, and third in ranks. The validity of metrics of word2vec models, especially that of emBLEU, was somewhat disappointing, probably owing to the domain difference between the training corpus and test corpus.
Abstract
The most recent language model in NLP, known as BERT, has numerous advantages over other language models. It ‘understands’ human language in a subword unit and can recognize rare words and out-of-vocabulary words. Furthermore, it considers syntax and context during the text-understanding process. We applied the evaluation metric “BERTscore” using this BERT model to evaluate human translation (HT). 120 translated texts were evaluated with five evaluation metrics: BLEU, METEOR, emBLEU, emMETEOR, as well as BERTscore and the result was compared with professional translators’ evaluation. The comparison examines the validity and reliability of these metrics, particularly the BERTscore, for future application for HT evaluation. BERTscore demonstrated a stable performance, taking first place in scores, and third in ranks. The validity of metrics of word2vec models, especially that of emBLEU, was somewhat disappointing, probably owing to the domain difference between the training corpus and test corpus.
참고문헌 (15)
- 정혜연, "번역자동평가에서 풀리지 않은 과제", 번역학연구, 21(1), pp.9~29 (2020), 한국번역학회[DOI]
- 김훈밀, "영어 어휘 다양성과 어휘 정교성이 한영번역에 미치는 영향", 번역학연구, 21(2), pp.43~65 (2020), 한국번역학회[DOI]
- 정혜연, "번역의 자동평가: 기계번역 평가를 인간번역 평가에 적용해보기", 통번역학연구, 22(4), pp.265~287 (2018), 통번역연구소[DOI]
- 정혜연, "임베딩을 활용한 인간번역의 자동평가 - 기계가 의미를 평가할 수 있을까", 통번역학연구, 25(3), pp.141~162 (2021), 통번역연구소[DOI]
- 정혜연, "인간번역 자동평가에서 정답자와 평가자가 다르다면", 독일언어문학, (93), pp.75~95 (2021), 한국독일언어문학회[DOI]
- 한국외대 번역평가인증 연구팀, "번역인증제도 (실무편)", 한국외대 통번역연구소 학술대회 <언어, 통번역의 평가 및 인증> 발표집, pp.23~33 (2016)
- Banerjee, S., "METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments", Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and, 2005, 65~72
- Chung, H. -Y., "Automatische Evaluation der Humanübersetzung: BLEU vs. METEOR", Lebende Sprachen, 65(1), pp.181~205 (2020)
- Devlin, J. Chang, M. -W., "BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding" (2019)[DOI]
- Flesch, R., "How to Write Plain English" (1981), Barnes &Noble
- Mikolov, T., "Distributed Representations of Words and Phrases and their Compositionality", Proceedings of Advances in Neural Information Processing Systems 26(NIPS 2013) (2013)
- Papineni, K., "BLEU: a Method for Automatic Evaluation of Machine Translation", Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pp.311~318 (2002)
- Servan, C., "Word2Vec vs DBnary: Augmenting METEOR using Vector Representations or Lexical Resources?", Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (2016)
- Wang, H., "Modification of Machine Translation Evaluation Metrics by Using word Embeddings", Proceedings of the Sixth Workshop on Hybrid Approaches to Translation, pp.33~41 (2016)
- Zhang, T., "BERTscore: Evaluating Text Generation with BERT", Conference paper at ICLR 2020 (2020)