Fine-Tuning-Free English Pronunciation Assessment System Using Multimodal Large Language Models
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
This study explores the performance and characteristics of Multimodal Large Language Models (MLLMs) in evaluating the English pronunciation of Korean speakers. While advancements in AI have enabled the development of various pronunciation assessment models with minimal data and fine-tuning, the limitation of requiring well-curated training data for specific purposes persisted. However, pre-trained Multimodal Large Language Models (MLLMs) have demonstrated the ability to perform tasks based on human instructions without additional training. This research investigates the effectiveness of using pre-trained MLLMs for pronunciation assessment without fine-tuning, comparing their evaluation results with those of human raters using the Pearson Correlation Coefficient (PCC). The results showed a high correlation of 0.798 for prosody scores, while articulation scores exhibited a lower correlation of 0.45. Despite scoring discrepancies between model and human raters, this study proposes a novel approach to pronunciation assessment without additional training, suggesting the potential for MLLMs to be widely applied as automated speaking assessment systems. These findings contribute to the growing body of research on AI-assisted language assessment and open new avenues for developing more efficient and adaptable pronunciation evaluation tools.
Abstract
This study explores the performance and characteristics of Multimodal Large Language Models (MLLMs) in evaluating the English pronunciation of Korean speakers. While advancements in AI have enabled the development of various pronunciation assessment models with minimal data and fine-tuning, the limitation of requiring well-curated training data for specific purposes persisted. However, pre-trained Multimodal Large Language Models (MLLMs) have demonstrated the ability to perform tasks based on human instructions without additional training. This research investigates the effectiveness of using pre-trained MLLMs for pronunciation assessment without fine-tuning, comparing their evaluation results with those of human raters using the Pearson Correlation Coefficient (PCC). The results showed a high correlation of 0.798 for prosody scores, while articulation scores exhibited a lower correlation of 0.45. Despite scoring discrepancies between model and human raters, this study proposes a novel approach to pronunciation assessment without additional training, suggesting the potential for MLLMs to be widely applied as automated speaking assessment systems. These findings contribute to the growing body of research on AI-assisted language assessment and open new avenues for developing more efficient and adaptable pronunciation evaluation tools.
참고문헌 (27)
- "Gpt-4 technical report" (2023)[URL]
- Ahn, T., "Automatic speech recognition(ASR)for the diagnosis of pronunciation of speech sound disorders in Korean children", Clinical Linguistics & Phonetics, pp.1~14 (2024)[DOI]
- "교육용 한국인의 영어 음성 데이터" (2023)[URL]
- Babu, A., "XLS-R: Self-Supervised Cross-Lingual Speech Representation Learning at Scale", Proc. Interspeech 2022, pp.2278~2282 (2022)[DOI]
- Baevski, A., "Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations", Advances in Neural Information Processing Systems, 33, pp.12449~12460 (2020)
- Botelho, C., "Macro-descriptors for Alzheimer’s disease detection using large language models", Proc. Interspeech 2024, pp.1975~1979 (2024)[DOI]
- Cho, Y., "A comparison of L1 and L2 speech phonetic posteriorgrams for applications in pronunciation training", Studies in Foreign Language Education, 35(1), pp.293~304 (2021)
- "Qwen2-Audio Technical Report" (2024)[URL]
- Feng, Y., "SED-MDD: Towards sentence dependent end-to-end mispronunciation detection and diagnosis", ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.3492~3496 (2020), IEEE
- Fouz-González, J., "Investigating English pronunciation: Trends and directions", pp.314~342 (2015)
- "Pronunciation Assessment with Multi-modal Large Language Models" (2024)[URL]
- Getman, Y., "Wav2vec2-Based Speech Rating System for Children with Speech Sound Disorder", Proc. Interspeech 2022, pp.3618~3622 (2022)[DOI]
- Heo, J., "LLM을 활용한 실전 어플리케이션 개발" (2024), 책만
- Hong, Y., "Evaluating score reliability of automatic English pronunciation assessment system for education", Studies in Foreign Language Education, 35(1), pp.91~104 (2021)
- "Qwen2-Audio-7B-Instruct" (2024)[URL]
- Jang, B. Y., "Computer-based fluency evaluation of English speaking tests for Koreans", Phonetics and Speech Sciences, 6(2), pp.9~20 (2014)
- "Automatic Pronunciation Assessment--A Review" (2023)[URL]
- Lee, J., "The Effectiveness of Second Language Pronunciation Instruction : A Meta-Analysis", Applied Linguistics, 36(3), pp.345~366 (2015)
- Manh Tien Anh, N., "Improving Speech Recognition with Prompt-based Contextualized ASR and LLM-based Re-predictor", Proc. Interspeech 2024, pp.737~741 (2024)[DOI]
- "Llama 3.2: Revolutionizing edge AI and vision with open, customizable models" (2024)[URL]
- "Hello GPT-4o" (2024)[URL]
- Peng, L., "A Study on Fine-Tuning wav2vec2. 0Model for the Task of Mispronunciation Detection and Diagnosis", Proc. Interspeech 2021, pp.4448~4452 (2021)[DOI]
- "Llama 2: Open foundation and fine-tuned chat models" (2023)[URL]
- Wagner, D., "Large Language Models for Dysfluency Detection in Stuttered Speech", Proc. Interspeech 2024, pp.5118~5122 (2024)[DOI]
- Yang, M., "Improving Mispronunciation Detection with Wav2vec2-Based Momentum Pseudo-Labeling for Accentedness and Intelligibility Assessment", Proc. Interspeech 2022, pp.4481~4485 (2022)[DOI]
- "Applying Wav2vec2.0 to Speech Recognition in Various Low-Resource Languages" (2021)[URL]
- Zuluaga-Gomez, J., "How Does Pre-Trained Wav2Vec 2.0Perform on Domain-Shifted ASR? An Extensive Benchmark on Air Traffic Control Communications", 2022 IEEE Spoken Language Technology Workshop (SLT) (2022), IEEE