Comparison of Hyper-Parameter Optimization Methods for Deep Neural Networks
0
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
Research into hyper parameter optimization (HPO) has recently revived with interest in models containing manyhyper parameters, such as deep neural networks. In this paper, we introduce the most widely used HPO methods,such as grid search, random search, and Bayesian optimization, and investigate their characteristics throughexperiments. The MNIST data set is used to compare results in experiments to find the best method that can beused to achieve higher accuracy in a relatively short time simulation. The learning rate and weight decay havebeen chosen for this experiment because these are the commonly used parameters in this kind of experiment.
Abstract
Research into hyper parameter optimization (HPO) has recently revived with interest in models containing manyhyper parameters, such as deep neural networks. In this paper, we introduce the most widely used HPO methods,such as grid search, random search, and Bayesian optimization, and investigate their characteristics throughexperiments. The MNIST data set is used to compare results in experiments to find the best method that can beused to achieve higher accuracy in a relatively short time simulation. The learning rate and weight decay havebeen chosen for this experiment because these are the commonly used parameters in this kind of experiment.
참고문헌 (9)
- M. Feurer, "Automated Machine Learning", pp.3~33 (2019), Springer
- J. Bergstra, "Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures", Proc. of the 30th International Conference on Machine Learning, 28, pp.115~123 (2013)
- J. Bergstra, "Random search for hyper-parameter optimization", Journal of machine learning research, 13, pp.281~305 (2012)[DOI]
- J. Snoek, "Practical bayesian optimization of machine learning algorithms", Advances in Neural Information Processing Systems, 25, pp.2951~2959 (2012)
- A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning, "Brochu, E.", UBC, ~, pp.TR-2009-23 (2009)
- "Combination of hyperband and bayesian optimization for hyperparameter optimization in deep learning" (2018)[URL]
- S. Falkner, "Bohb:Robust and efficient hyperparameter optimization at scale", Proceedings of Machine Learning Research, 80, pp.1437~1446 (2018)
- "“Practical guide to hyperparameters optimization for deep learning models,” Deep Learning" (2018)[URL]
- Y. LeCun, "Gradient based learning applied to document recognition", Proceedings of the IEEE, 86(11), pp.2278~2324 (1998)