Evaluating Score Reliability of Automatic English Pronunciation Assessment System for Education
1
조회
0
다운로드
5
인용
0
참고문헌
키워드
초록
The purpose of this study was to evaluate the pronunciation score reliability of an automatic pronunciation assessment system for education, SpeechPro, a commercially released and patented system but without a score reliability test. So it is necessary to ensure the commercial system’s reliability. The method is to measure score agreement between SpeechPro and human raters. The database used is a paid English speech corpus of the native speakers and non-native speakers with score annotations of the three English raters. First, the inter-rater agreement was measured, and then the agreement between SpeechPro’s scores and the raters’ average scores were measured. The following 5 metrics were used: Pearson correlation coefficient, standardized mean difference, quadratic weighted kappa, exact percentage agreement, and 1-point adjacent percentage agreement. The results are that human-machine agreement is significantly identical to human-human agreement according to all the metrics used, proving the score reliability of SpeechPro. This provides a logical justification required before the comparison with other automatic pronunciation assessment systems.
Abstract
The purpose of this study was to evaluate the pronunciation score reliability of an automatic pronunciation assessment system for education, SpeechPro, a commercially released and patented system but without a score reliability test. So it is necessary to ensure the commercial system’s reliability. The method is to measure score agreement between SpeechPro and human raters. The database used is a paid English speech corpus of the native speakers and non-native speakers with score annotations of the three English raters. First, the inter-rater agreement was measured, and then the agreement between SpeechPro’s scores and the raters’ average scores were measured. The following 5 metrics were used: Pearson correlation coefficient, standardized mean difference, quadratic weighted kappa, exact percentage agreement, and 1-point adjacent percentage agreement. The results are that human-machine agreement is significantly identical to human-human agreement according to all the metrics used, proving the score reliability of SpeechPro. This provides a logical justification required before the comparison with other automatic pronunciation assessment systems.
참고문헌 (23)
- Beatty, K., "Teaching and researching computer-assisted language learning" (2003), Person Education
- "The CMU pronouncing dictionary" (2015)[URL]
- Chen, H., "Automated essay scoring by maximizing human-machine agreement", Proce dings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp.1741~1752 (2013)
- 정현성, "한국인 영어 학습자의 발음 정확성 자동 측정방법에 대한 연구", 언어와언어학, (42), pp.165~196 (2008), 언어연구소
- Cincarek, T., "Automatic pronunciation scoring of words and sentences independent from the non-native’s first language", Computer Speech &Language, 23(1), pp.65~88 (2009)
- Dahl, G. E., "Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition", IEEE Transactions on audio, speech, and language processing, 20(1), pp.30~42 (2011)
- Hanna, L., "An investigation of Modern Foreign Language (MFL)teachers and their cognitions of Computer Assisted Language Learning (CALL) amid the COVID-19 health pandemic", ~ (2020)
- Hedges, L. V., "A standardized mean difference effect size for single case designs", Research Synthesis Methods, 3(3), pp.224~239 (2012)
- Herron, D., "Automatic localization and diagnosis of pronunciation errors for second-language learners of English", Paper presented at the 6th European Conference on Speech Communication and Technology (1999)
- Kim, I. S., "Automatic speech recognition: Reliability and pedagogical implications for teaching pronunciation", Educational Technology & Society, 9(1), pp.322~334 (2006)
- 김미란, "영어학습자 발음의 리듬향상과 관련된 언어요인 연구", 외국어교육연구, 34(1), pp.237~261 (2020), 외국어교육연구소[DOI]
- 김태식, "영어발음교육의 PBL적 접근: 영어교육과 1학년의 사례연구", 외국어교육연구, 33(4), pp.325~354 (2019), 외국어교육연구소[DOI]
- Performance of automated speech scoring on different low- to medium-entropy item types for low-proficiency English learners, "Loukina, A.", ETS, 1~17, pp.ETS Research Report Series, 2017(1) (2017)
- Menzel, M., "Interactive pronunciation training", ReCALL, 13(1), pp.67~78 (2001)
- Panayotov, V., "Librispeech: an asr corpus based on public domain audio books", Paper presented at 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP) (2015), IEEE
- Povey, D., "The Kaldi speech recognition toolkit", Paper presented at IEEE 2011 workshop on automatic speech recognition and understanding (2011), IEEE Signal Processing Society
- Prafianto, H., "Improving human scoring of prosody using parametric speech synthesis", Speech Communication, 111, pp.14~21 (2019)
- "King-ASR-704: Accented English Pronunciation Evaluation Corpus (Word level)" (2020)[URL]
- Witt, S., "Language Teaching & Language Technology", pp.25~35 (1998), Swets & Zeitlinger
- Wang, Z., "Monitoring the performance of human and automated scores for spoken responses", Language Testing, 35(1), pp.101~120 (2018)
- 윤원희, "영어 발음 평가에서 한국인 평가자 사이의 평가 차이", 외국어교육연구, 23(2), pp.85~103 (2009), 외국어교육연구소[DOI]
- Tafazoli, D., "Technology-based review on Computer-Assisted Language Learning: A chronological perspective", Pixel-Bit: Revista de Medios y Educación, 54, pp.29~43 (2019)
- 미디어젠(주), "특허출원제40-2020-0157130", 특허청, ~ (2020)