Hyperparameter experiments on end-to-end automatic speech recognition
1
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
End-to-end (E2E) automatic speech recognition (ASR) has achieved promising performance gains with the introduced self-attention network, Transformer. However, due to training time and the number of hyperparameters, finding the optimal hyperparameter set is computationally expensive. This paper investigates the impact of hyperparameters in the Transformer network to answer two questions: which hyperparameter plays a critical role in the task performance and training speed. The Transformer network for training has two encoder and decoder networks combined with Connectionist Temporal Classification (CTC). We have trained the model with Wall Street Journal (WSJ) SI-284 and tested on devl93 and eval92. Seventeen hyperparameters were selected from the ESPnet training configuration, and varying ranges of values were used for experiments. The result shows that “num blocks” and “linear units” hyperparameters in the encoder and decoder networks reduce Word Error Rate (WER) significantly. However, performance gain is more prominent when they are altered in the encoder network. Training duration also linearly increased as “num blocks” and “linear units” hyperparameters’ values grow. Based on the experimental results, we collected the optimal values from each hyperparameter and reduced the WER up to 2.9/1.9 from dev93 and eval93 respectively.
Abstract
End-to-end (E2E) automatic speech recognition (ASR) has achieved promising performance gains with the introduced self-attention network, Transformer. However, due to training time and the number of hyperparameters, finding the optimal hyperparameter set is computationally expensive. This paper investigates the impact of hyperparameters in the Transformer network to answer two questions: which hyperparameter plays a critical role in the task performance and training speed. The Transformer network for training has two encoder and decoder networks combined with Connectionist Temporal Classification (CTC). We have trained the model with Wall Street Journal (WSJ) SI-284 and tested on devl93 and eval92. Seventeen hyperparameters were selected from the ESPnet training configuration, and varying ranges of values were used for experiments. The result shows that “num blocks” and “linear units” hyperparameters in the encoder and decoder networks reduce Word Error Rate (WER) significantly. However, performance gain is more prominent when they are altered in the encoder network. Training duration also linearly increased as “num blocks” and “linear units” hyperparameters’ values grow. Based on the experimental results, we collected the optimal values from each hyperparameter and reduced the WER up to 2.9/1.9 from dev93 and eval93 respectively.
참고문헌 (20)
- Chang, X., "End-to-end multi-speaker speech recognition with transformer", Proceedings of the ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2020)
- Gale, W. A., "Good-turing frequency estimation without tears", Journal of Quantitative Linguistics, 2(3), pp.217~237 (1995)
- Graves, A., "Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks", Proceedings of the 23rd International Conference on Machine Learning, pp.369~376 (2006)
- "Modified kneser-ney smoothing of n-gram models (RIACS Technical Report 00.07)" (2000)[URL]
- "KoSpeech: open-source toolkit for end-to-end Korean speech recognition" (2020)[URL]
- "Adam: a method for stochastic optimization" (2014)[URL]
- Koutsoukas, A., "Deeplearning : investigating deep neural networks hyper-parameters and comparison of performance to shallow methods for modeling bioactivity data", Journal of Cheminformatics, 9(1), pp.1~13 (2017)
- Lakomkin, E., "Dorporating end-to-end speech recognition models for sentiment analysis", Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), pp.7976~7982 (2019)
- LeCun, Y. A., "Neural networks: tricks of the trade, Vol. 7700", pp.9~48 (2012), Springer
- Miao, H., "Transformer-based online CTC", attention end-to-end speech recognition architecture, 2020, 6084~6088
- Nakatani, T., "Improving transformer-based end-to-end speech recognition with connectionist temporal classification and language model integration", Proceedings of the Interspeech 2019 (2019)
- Okewu, E., "Experimental comparison of stochastic optimizers in deep learning", Proceedings of the International Conference on Computational Science and its Applications, pp.704~715 (2019)
- Popel, M., "Training tips for the transformer model", The Prague Bulletin of Mathematical Linguistics, 110(1), pp.43~70 (2018)
- "Attention is all you need" (2017)[URL]
- "Semantic mask for transformer based end-to-end speech recognition" (2019)[URL]
- Watanabe, S., "The 2020 ESPnet update: new features, broadened applications, performance improvements, and future plans", arXiv, arXiv:2012.13006, ~ (2020)
- "Espnet: end-to-end speech processing toolkit" (2018)[URL]
- Watanabe, S., "Hybrid CTC", attention architecture for end-to-end speech recognition, 2017, pp.11(8), 1240~1253
- Wei, C., "How to build a chatbot:chatbot framework and its capabilities", Proceedings of the 201810th International Conference on Machine Learning and Computing, pp.369~373 (2018)
- "Large batch optimization for deep learning:training bert in 76 minutes" (2019)[URL]