인간번역의 자동평가에서 번역 텍스트 길이가 평가 타당성에 미치는 영향
Effect of Text Length on the Validity of Automatic Evaluation of Human Translation
0
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
This study explores the influence of the length of human-translated text on the validity of automatic evaluations. To this end, the Brevity Penalty (BP) was replaced with the Length Penalty (LP) to better align the evaluation methodology with the characteristics of human translation. The study examined if this modification improved evaluation validity. The impact of the BP on the total score was ascertained via the coefficient of determination (R²) of the regression analysis. Validity was further investigated by comparing the correlation coefficient of the similarity between BLEU and LP-applied BLEU (lBLEU) scores and human evaluation scores. The findings indicated an overall improvement in validity, and an increase in the coefficient of determination. The qualitative analysis revealed that students were more likely to lose points for longer translations when compared to the three reference translations, particularly in the case of extensive, informative original texts on unfamiliar topics. Therefore, the length of the human-translated text significantly influences the validity of the automatic evaluation, and substituting BP with LP positively improved this validity. The implications of this study can potentially extend to educational scenarios and the utilization of automated assessments.
Abstract
This study explores the influence of the length of human-translated text on the validity of automatic evaluations. To this end, the Brevity Penalty (BP) was replaced with the Length Penalty (LP) to better align the evaluation methodology with the characteristics of human translation. The study examined if this modification improved evaluation validity. The impact of the BP on the total score was ascertained via the coefficient of determination (R²) of the regression analysis. Validity was further investigated by comparing the correlation coefficient of the similarity between BLEU and LP-applied BLEU (lBLEU) scores and human evaluation scores. The findings indicated an overall improvement in validity, and an increase in the coefficient of determination. The qualitative analysis revealed that students were more likely to lose points for longer translations when compared to the three reference translations, particularly in the case of extensive, informative original texts on unfamiliar topics. Therefore, the length of the human-translated text significantly influences the validity of the automatic evaluation, and substituting BP with LP positively improved this validity. The implications of this study can potentially extend to educational scenarios and the utilization of automated assessments.
참고문헌 (21)
- 정혜연, "번역의 자동평가: 기계번역 평가를 인간번역 평가에 적용해보기", 통번역학연구, 22(4), pp.265~287 (2018), 통번역연구소[DOI]
- 정혜연, "번역자동평가에서 풀리지 않은 과제", 번역학연구, 21(1), pp.9~29 (2020), 한국번역학회[DOI]
- 정혜연, "임베딩을 활용한 인간번역의 자동평가 - 기계가 의미를 평가할 수 있을까", 통번역학연구, 25(3), pp.141~162 (2021), 통번역연구소[DOI]
- 정혜연, "인간 번역평가에서 재현도(recall)의 중요성", 번역학연구, 23(1), pp.81~100 (2022), 한국번역학회[DOI]
- 정혜연, "버트(BERT)를 활용한 인간번역의 자동평가: 여러 모델의 성능 비교 및 활용 가능성", 통번역학연구, 26(4), pp.117~137 (2022), 통번역연구소[DOI]
- 한국외대 번역평가인증 연구팀, "번역인증제도 (실무편)", 한국외국어대학교통번역연구소 학술대회, pp.23~35 (2016)
- Banerjee, S., "METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments", Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and, 2005, 65~72
- Chen, B., "A Systematic Comparison of Smoothing Techniques for Sentence-level BLEU", Proceedings of the Ninth Workshop on Statistical Machine Translation, pp.362~367 (2014)
- Chung, H. Y., "Automatische evaluation der Humanübersetzung:BLEU vs. METEOR", Lebende Sprachen, 65(1), pp.181~205 (2020)
- Dancey, C. P., "Statistics without maths for psychology" (2017), Pearson
- Fleckenstein, J., "Is a long essay always a good essay? The effect of text length on writing assessment", Frontiers in psychology, 11, pp.562462~ (2020)
- Han, A. L., "LEPOR: A robust evaluation metric for machine translation with augmented factors", Proceedings of COLING 2012: Posters, pp.441~450 (2012)
- Han, C., "Interpreting quality assessment re-imagined : The synergy between human and machine scoring", Interpreting and Society, 1(1), pp.70~90 (2021)
- Kim, J., "Generative Multi-Turn Chatbot Using Generative Adversarial Network", Proceedings in Annual Conference on Human and Language Technology, 0, pp.25~30 (2018)
- Lu, X., "Automatic assessment of spoken-language interpreting based on machine-translation evaluation metrics : A multi-scenario exploratory study", Interpreting, 25(1), pp.109~143 (2023)
- Papineni, K., "Bleu: a method for automatic evaluation of machine translation", Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp.311~318 (2002)
- Vela, M., "Human translation evaluation and its coverage by automatic scores", Automatic and Manual Metrics for Operational Translation Evaluation Workshop Programme, pp.19~ (2014)
- Yuan, Y., "Mobil: A hybrid feature set for automatic human translation quality assessment", Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pp.3663~3670 (2016)
- "Flesch-Kincaid Readability"[URL]
- "mecab-ko-dic-2.1.1-20180720 배포합니다" (2018)[URL]
- "Source code for nltk.translate.bleu_score"[URL]