딥러닝을 위한 비단조 활성화 함수
Non-monotonic activation function for deep learning
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
The activation function significantly affects the performance of neural networks. Among the numerous functions, the Rectified Linear Unit(ReLU) is widely used in many deep learning applications owing to its simplicity and performance. This study proposes a new nonlinear activation function derived from logarithmic and hyperbolic tangent functions. It exhibits the following distinct characteristics: 1) If the input is greater than 0, then the output is the same as the input, 2) if the input is approximately 0, then the output exhibits non-linear characteristics, and 3) if the input is negative infinity, then the output has a value of approximately zero. Simulation results show that the proposed activation function surpasses the ReLU, Mish, and Power Function Linear Units in terms of classification accuracy. In particular, when applied to the CIFAR-10 classification using the VGG19 network, it increases the accuracy by approximately 1%.
Abstract
The activation function significantly affects the performance of neural networks. Among the numerous functions, the Rectified Linear Unit(ReLU) is widely used in many deep learning applications owing to its simplicity and performance. This study proposes a new nonlinear activation function derived from logarithmic and hyperbolic tangent functions. It exhibits the following distinct characteristics: 1) If the input is greater than 0, then the output is the same as the input, 2) if the input is approximately 0, then the output exhibits non-linear characteristics, and 3) if the input is negative infinity, then the output has a value of approximately zero. Simulation results show that the proposed activation function surpasses the ReLU, Mish, and Power Function Linear Units in terms of classification accuracy. In particular, when applied to the CIFAR-10 classification using the VGG19 network, it increases the accuracy by approximately 1%.
참고문헌 (9)
- Aurélien Géron, "Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow" (2019), O'Reilly Media, Inc
- Nair, Vinod, "Rectified linear units improve restricted boltzmann machines", Proceedings of the 27th international conference on machine learning(ICML-10), pp.807~814 (2010)
- Maas, Andrew L, "Rectifier nonlinearities improve neural network acoustic models", Proc. icml, 30(1), pp.3~ (2013)
- Klambauer, Gunter, "Self-normalizing neural networks", 31st Conference on Neural Information Processing Systems (2017)
- Howard, Andrew, "Searching for mobilenetv3", Proceedings of the IEEE, 2019, 1314~1324
- Zhu, Meng, "PFLU and FPFLU : Two novel non-monotonic activation functions in convolutional neural networks", Neurocomputing, 429, pp.110~117 (2021)
- Krizhevsky, A., "Learning multiple layers of features from tiny images", pp.2009