Zero-shot voice conversion with HuBERT
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
This study introduces an innovative model for zero-shot voice conversion that utilizes the capabilities of HuBERT. Zero-shot voice conversion models can transform the speech of one speaker to mimic that of another, even when the model has not been exposed to the target speaker's voice during the training phase. Comprising five main components (HuBERT, feature encoder, flow, speaker encoder, and vocoder), the model offers remarkable performance across a range of scenarios. Notably, it excels in the challenging unseen-to-unseen voice-conversion tasks. The effectiveness of the model was assessed based on the mean opinion scores and similarity scores, reflecting high voice quality and similarity to the target speakers. This model demonstrates considerable promise for a range of real-world applications demanding high-quality voice conversion. This study sets a precedent in the exploration of HuBERT-based models for voice conversion, and presents new directions for future research in this domain. Despite its complexities, the robust performance of this model underscores the viability of HuBERT in advancing voice conversion technology, making it a significant contributor to the field.
Abstract
This study introduces an innovative model for zero-shot voice conversion that utilizes the capabilities of HuBERT. Zero-shot voice conversion models can transform the speech of one speaker to mimic that of another, even when the model has not been exposed to the target speaker's voice during the training phase. Comprising five main components (HuBERT, feature encoder, flow, speaker encoder, and vocoder), the model offers remarkable performance across a range of scenarios. Notably, it excels in the challenging unseen-to-unseen voice-conversion tasks. The effectiveness of the model was assessed based on the mean opinion scores and similarity scores, reflecting high voice quality and similarity to the target speakers. This model demonstrates considerable promise for a range of real-world applications demanding high-quality voice conversion. This study sets a precedent in the exploration of HuBERT-based models for voice conversion, and presents new directions for future research in this domain. Despite its complexities, the robust performance of this model underscores the viability of HuBERT in advancing voice conversion technology, making it a significant contributor to the field.
참고문헌 (30)
- Baevski, A., "wav2vec 2.0: A framework for self-supervised learning of speech representations", Proceedings of the Advances in Neural Information Processing Systems. Online Conference (2020)
- "Hi-fi multi-speaker English TTS dataset" (2021)[URL]
- Buduma, N., "Fundamentals of deep learning" (2022), O'Reilly Media
- "Bert : Pre-training of deep bidirectional transformers for language understanding" (2018)[URL]
- "Generating sequences with recurrent neural networks" (2013)[URL]
- Hochreiter, S., "Long short-term memory", Neural Computation, 9(8), pp.1735~1780 (1997)
- Hsu, W. N., "HuBERT: Self-supervised speech representation learning by masked prediction of hidden units", IEEE, 2021, pp.29, 3451~3460
- Ioffe, S., "Batch normalization: Accelerating deep network training by reducing internal covariate shift", Proceedings of the International Conference on Mmachine Learning, pp.448~456 (2015)
- "On large-batch training for deep learning : Generalization gap and sharp minima" (2016)[URL]
- Kim, J., "Glow-TTS: A generative flow for text-to-speech via monotonic alignment search", Poceedings of the Advances in Neural Information Processing Systems. Online Conference (2020)
- Kim, J., "Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech", Proceedings of the International Conference on Machine Learning, pp.5530~5540 (2021)
- "Adam : A method for stochastic optimization" (2014)[URL]
- Kingma, D. P., "Glow: Generative flow with invertible 1x1 convolutions", Proceedings of the Advances in Neural Information Processing Systems (2018)
- Kong, J., "HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis", Proceedings of the Advances in Neural Information Processing Systems. Online Conference (2020)
- "Cutting down on prompts and parameters : Simple few-shot learning with language models" (2021)[URL]
- McFee, B., "librosa: Audio and music signal analysis in python", Proceedings of the 14th Python in Science Conference (2015)
- Palatucci, M., "Zero-shot learning with semantic output codes", Proceedings of the Advances in Neural Information Processing Systems (2009)
- Qian, K., "AutoVC: Zero-shot voice style transfer with only autoencoder loss", Proceedings of the International Conference on Machine Learning, pp.5210~5219 (2019)
- Rezende, D., "Variational inference with normalizing flows", Proceedings of the International Conference on Machine Learning, pp.1530~1538 (2015)
- "RVC-Project/Retrieval-based-Voice- ConversionWebUI" (2023)[URL]
- "wav2vec : Unsupervised pre-training for speech recognition" (2019)[URL]
- Schuster, M., "Bidirectional recurrent neural networks", IEEE Transactions on Signal Processing, 45(11), pp.2673~2681 (1997)
- "NaturalSpeech 2 : Latent diffusion models are natural and zero-shot speech and singing synthesizers" (2023)[URL]
- Sisman, B., "An overview of voice conversion and its challenges: From statistical modeling to deep learning", IEEE, 2021, pp.29, 132~157
- Sohl-Dickstein, J., "Deep unsupervised learning using nonequilibrium thermodynamics", Proceedings of the International Conference on Machine Learning, pp.2256~2265 (2015)
- "svc-develop-team/so-vits-svc" (2023)[URL]
- van Niekerk, B., "A comparison of discrete and soft speech units for improved voice conversion", Proceedings of the 2022 IEEE International Conference on Acoustics, Speech and Signal Processing, pp.6562~6566 (2022)
- Vaswani, A., "Attention is all you need", Proceedings of the Advances in Neural Information Processing Systems (2017)
- Wester, M., "Analysis of the voice conversion challenge 2016 evaluation results", Proceedings of the Interspeech, pp.1637~1641 (2016)
- CSTR VCTK corpus: English multi-speaker corpus for CSTR voice cloning toolkit, "Yamagishi, J.", University of Edinburgh. The Centre for Speech Technology Research (CSTR), ~ (2019)