통역 자동 평가의 가능성과 한계 고찰- MTQE 적용 사례를 중심으로-
Automated Interpreting Assessment: Leveraging MTQE to Evaluate Interpreting Accuracy
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
This study investigated the potential benefits and limitations of automated interpreting assessment by reviewing various approaches, including machine translation (MT) quality metrics and quality estimation (QE), followed by an experimental application of MTQE to a small-scale dataset of English-Korean consecutive interpretations by three interpreting students. CometKiwi, one of the state-of-the-art MTQE methods that show strong evaluation performance without requiring reference translations, was employed to compare automated evaluation with human evaluation. The findings reveal that automated evaluations exhibited a strong correlation with human evaluations and achieved complete agreement in ranking interpreting outputs, confirming the feasibility of QE-based automated evaluation for interpretations. At both the text and segment levels, higher-quality interpreting outputs showed greater alignment between human and automated evaluations, while lower-quality outputs tended to receive relatively higher scores from the QE model, highlighting discrepancies with human evaluations. It was noted that CometKiwi returned scores above zero for uninterpreted segments, possibly overestimating the final overall quality scores. The study suggests that automated evaluation scores could serve as useful resources for interpreting educators and students when an independent reference source is needed to support human evaluation.
Abstract
This study investigated the potential benefits and limitations of automated interpreting assessment by reviewing various approaches, including machine translation (MT) quality metrics and quality estimation (QE), followed by an experimental application of MTQE to a small-scale dataset of English-Korean consecutive interpretations by three interpreting students. CometKiwi, one of the state-of-the-art MTQE methods that show strong evaluation performance without requiring reference translations, was employed to compare automated evaluation with human evaluation. The findings reveal that automated evaluations exhibited a strong correlation with human evaluations and achieved complete agreement in ranking interpreting outputs, confirming the feasibility of QE-based automated evaluation for interpretations. At both the text and segment levels, higher-quality interpreting outputs showed greater alignment between human and automated evaluations, while lower-quality outputs tended to receive relatively higher scores from the QE model, highlighting discrepancies with human evaluations. It was noted that CometKiwi returned scores above zero for uninterpreted segments, possibly overestimating the final overall quality scores. The study suggests that automated evaluation scores could serve as useful resources for interpreting educators and students when an independent reference source is needed to support human evaluation.