한국어 번역평가 참조 방향에 관한 고찰: 기계학습 분류 성능 평가지표를 활용하여
Direction of Translation Evaluation: Using Machine Learning Classification Performance Metrics
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
This study proposes a method of inferring the preferred evaluation direction of humans in assessing translations. By adjusting the precision and recall ratio in machine learning classification, the researchers aimed to match human evaluations. They varied the ratio of the widely used automatic evaluation metric, BLEU, and compared the results to human evaluations. The findings reveal that a precision:recall ratio of 10:0 closely resembles human evaluation for translators, while a ratio of 5:5 was the closest for texts, and a ratio of 9:1 for total evaluation. This experiment is significant as it attempts to predict the evaluation direction in human translation assessments using machine indicators. It has the potential to contribute to establishment of standards and evaluation indicators in the translation field, which have been subjects of discussion in academia and practice.
Abstract
This study proposes a method of inferring the preferred evaluation direction of humans in assessing translations. By adjusting the precision and recall ratio in machine learning classification, the researchers aimed to match human evaluations. They varied the ratio of the widely used automatic evaluation metric, BLEU, and compared the results to human evaluations. The findings reveal that a precision:recall ratio of 10:0 closely resembles human evaluation for translators, while a ratio of 5:5 was the closest for texts, and a ratio of 9:1 for total evaluation. This experiment is significant as it attempts to predict the evaluation direction in human translation assessments using machine indicators. It has the potential to contribute to establishment of standards and evaluation indicators in the translation field, which have been subjects of discussion in academia and practice.
참고문헌 (30)
- 강승민, "증상 서술에 의한 Covid-19 감염 분류", Proceedings of KIIT Conference, pp.598~601 (2020)
- 권철민, "파이썬 머신러닝 완벽 가이드" (2020), 위키북스
- 박도준, "문법 정확도 평가(GAE): 기계 번역 모델의 정량화된 정성 평가", 정보과학회논문지, 49(7), pp.514~520 (2022), 한국정보과학회[DOI]
- 이경건, "인공지능 기반 자동평가의 현재와 미래: 서술형 문항에 관한 문헌 고찰과 그 너머", 교육공학연구, 36(2), pp.353~382 (2020), 한국교육공학회[DOI]
- 이선복, "기계학습의 소개: 예측모형을 위한 지도학습 이론을 중심으로", 교육과학연구, 53(3), pp.1~43 (2022), 교육과학연구소
- 정혜연, "임베딩을 활용한 인간번역의 자동평가 - 기계가 의미를 평가할 수 있을까", 통번역학연구, 25(3), pp.141~162 (2021), 통번역연구소[DOI]
- 정혜연, "인간 번역평가에서 재현도(recall)의 중요성", 번역학연구, 23(1), pp.81~100 (2022), 한국번역학회[DOI]
- 정혜연, "버트(BERT)를 활용한 인간번역의 자동평가: 여러 모델의 성능 비교 및 활용 가능성", 통번역학연구, 26(4), pp.117~137 (2022), 통번역연구소[DOI]
- 최승권, "자동통번역 시스템의 언어 현상별 자동 평가", 한국정보처리학회 학술대회논문집, 26(2), pp.1041~1044 (2019)
- 한국외대 번역평가인증 연구팀, "번역인증제도(실무편)", 한국외대 통번역연구소 학술대회 언어, 통번역의 평가 및 인증 발표집, pp.23~33 (2016)
- Banerjee, S, "METEOR: An automatic metric for MT evaluation with improved correlation with human judgments", Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and, 2005, 65~72
- Chung, H. Y., "Automatische evaluation der Humanübersetzung: BLEU vs. METEOR", Lebende Sprachen, 65(1), pp.181~205 (2020)
- Dancey, C. P., "Statistics without maths for psychology", pp.632~ (2017), Pearson
- Devlin, J., "Bert: Pre-training of deep bidirectional transformers for language understanding", Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human LAnguage Technologies, 1, pp.4171~4186 (2019)
- Flesch, R., "A new readability yardstick", Journal of applied psychology, 32(3), pp.221~ (1948)
- Han, A. L., "LEPOR: A robust evaluation metric for machine translation with augmented factors", Proceedings of COLING 2012: Posters, pp.441~450 (2012)
- Han, L., "Machine translation evaluation resources and methods: A survey", ~ (2016)
- Hanna, M., "A fine-grained analysis of BERTScore", Proceedings of the Sixth Conference on Machine Translation, pp.507~517 (2021)
- Kincaid, J. P., "Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel", ~ (1975)
- Melamed, I. D., "Precision and recall of machine translation", Companion Volume of the Proceedings of HLT-NAACL 2003- Short Papers, pp.61~63 (2003)
- Lavie, A., "The significance of recall in automatic metrics for MT evaluation", Machine Translation: From Real Users to Research: 6th Conference of the Association for Machine Translation in the Americas, AMTA 2004, pp.134~143 (2004), Springer Berlin Heidelberg
- Lin, C. Y., "Automatic evaluation of summaries using n-gram co-occurrence statistics", Proceedings of the 2003 human language technology conference of the North American chapter of the association for computational linguistics, pp.150~157 (2003)
- Lu, X., "Automatic assessment of spoken-language interpreting based on machine-translation evaluation metrics: A multi-scenario exploratory study", Interpreting, 25(1), pp.109~143 (2023)
- Papineni, K., "Bleu: a method for automatic evaluation of machine translation", Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp.311~318 (2002)
- Vela, M., "Human translation evaluation and its coverage by automatic scores", Automatic and Manual Metrics for Operational Translation Evaluation Workshop Programme, pp.19~ (2014)
- Zhang, T., "Bertscore: Evaluating text generation with bert", ~ (2019)
- Zerva, C., "Findings of the wmt 2022 shared task on quality estimation", Proceedings of the Seventh Conference on Machine Translation (WMT), pp.69~99 (2022)
- "Flesch-kincaid 이독성 지수 계산"[URL]
- "형태소 분석기 mecab.ko(버전: mecab-ko-dic-2.1.1 – 20180720)"[URL]
- "NLTK-BLEU scoring codes"[URL]