Exploring the feasibility of fine-tuning large-scale speech recognition models for domain-specific applications: A case study on Whisper model and KsponSpeech dataset
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
This study investigates the fine-tuning of large-scale Automatic Speech Recognition (ASR) models, specifically OpenAI’s Whisper model, for domain-specific applications using the KsponSpeech dataset. The primary research questions address the effectiveness of targeted lexical item emphasis during fine-tuning, its impact on domain-specific performance, and whether the fine-tuned model can maintain generalization capabilities across different languages and environments. Experiments were conducted using two fine-tuning datasets: Set A, a small subset emphasizing specific lexical items, and Set B, consisting of the entire KsponSpeech dataset. Results showed that fine-tuning with targeted lexical items increased recognition accuracy and improved domain-specific performance, with generalization capabilities maintained when fine-tuned with a smaller dataset. For noisier environments, a trade-off between specificity and generalization capabilities was observed. This study highlights the potential of fine-tuning using minimal domain-specific data to achieve satisfactory results, emphasizing the importance of balancing specialization and generalization for ASR models. Future research could explore different fine-tuning strategies and novel technologies such as prompting to further enhance large-scale ASR models’ domain-specific performance.
Abstract
This study investigates the fine-tuning of large-scale Automatic Speech Recognition (ASR) models, specifically OpenAI’s Whisper model, for domain-specific applications using the KsponSpeech dataset. The primary research questions address the effectiveness of targeted lexical item emphasis during fine-tuning, its impact on domain-specific performance, and whether the fine-tuned model can maintain generalization capabilities across different languages and environments. Experiments were conducted using two fine-tuning datasets: Set A, a small subset emphasizing specific lexical items, and Set B, consisting of the entire KsponSpeech dataset. Results showed that fine-tuning with targeted lexical items increased recognition accuracy and improved domain-specific performance, with generalization capabilities maintained when fine-tuned with a smaller dataset. For noisier environments, a trade-off between specificity and generalization capabilities was observed. This study highlights the potential of fine-tuning using minimal domain-specific data to achieve satisfactory results, emphasizing the importance of balancing specialization and generalization for ASR models. Future research could explore different fine-tuning strategies and novel technologies such as prompting to further enhance large-scale ASR models’ domain-specific performance.
참고문헌 (21)
- Baevski, A., "wav2vec 2.0: A framework for self-supervised learning of speech representations", Proceedings of the Advances in Neural Information Processing Systems, 2, pp.12449~12460 (2020)
- Bang, J. U., "KsponSpeech : Korean spontaneous speech corpus for automatic speech recognition", Applied Sciences, 10(19), pp.6936~ (2020)
- "SpeechPrompt: An exploration of prompt tuning on generative spoken language model for speech processing tasks" (2022)[URL]
- Chen, S., "WavLM : Large-scale self-supervised pre-training for full stack speech processing", IEEE Journal of Selected Topics in Signal Processing, 16(6), pp.1505~1518 (2022)
- "Bert: Pre-training of deep bidirectional transformers for language understanding" (2018)[URL]
- "Conformer: Convolution-augmented transformer for speech recognition" (2020)[URL]
- Guo, P., "Recent developments on espnet toolkit boosted by conformer", Proceedings of the ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.5874~5878 (2021)
- Hsu, W. N., "HuBERT: Self-supervised speech representation learning by masked prediction of hidden units", IEEE, 2021, pp.29, 3451~3460
- Kim, K., "E-branchformer : Branchformer with enhanced merging for speech recognition", Proceedings of the 2022 IEEE Spoken Language Technology Workshop (SLT), pp.84~91 (2023)
- "Adam: A method for stochastic optimization" (2014)[URL]
- "Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing" (2021)[URL]
- Mohamed, A., "Self-supervised speech representation learning : A review", IEEE Journal of Selected Topics in Signal Processing, 16(6), pp.1179~1210 (2022)
- Panayotov, V., "Librispeech: an ASR corpus based on public domain audio books", Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.5206~5210 (2015)
- "Prompting the hidden talent of web-scale speech models for zero-shot task generalization" (2023)[URL]
- "A comparative study on E-branchformer vs conformer in speech recognition, translation, and understanding tasks" (2023)[URL]
- "Scaling speech technology to 1,000+ languages" (2023)[URL]
- Radford, A., "Robust speech recognition via large-scale weak supervision", Proceedings of the 40th International Conference on Machine Learning, pp.28492~28518 (2023)
- "Comparison of multilingual self-supervised and weakly-supervised speech pre-training for adaptation to unseen languages" (2023)[URL]
- Vaswani, A., "Attention is all you need", Proceedings of the Advances in Neural Information Processing Systems (2017)
- "Huggingface’s transformers: State-of-the-art natural language processing" (2019)[URL]
- "Google usm : Scaling automatic speech recognition beyond 100 languages" (2023)[URL]