Probabilistic Target Encoding of Neural Networks with Softmax Output Nodes
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
When training neural networks with softmax outputs, one-hot target encoding is commonly used due to its simplicity. This strategy does not incorporate the probability that an input sample belongs to a certain class but adopts “one” or “zero” as the desired output values of neural networks. Instead of the most prevalent one-hot encoding, this paper proposes a probabilistic target encoding to prevent the overfitting of neural networks to training samples. This effect brings about the accuracy improvement of test samples. We demonstrated the effectiveness of the proposed target encoding through simulations of multi-layer perceptrons and convolutional neural networks for various classification problems such as handwritten-digit recognition, isolated-word recognition, image classification, and object recognition tasks. The simulation results show that the proposed probabilistic target encoding is superior to one-hot encoding as it prevents the overfitting of neural networks to training samples.
Abstract
When training neural networks with softmax outputs, one-hot target encoding is commonly used due to its simplicity. This strategy does not incorporate the probability that an input sample belongs to a certain class but adopts “one” or “zero” as the desired output values of neural networks. Instead of the most prevalent one-hot encoding, this paper proposes a probabilistic target encoding to prevent the overfitting of neural networks to training samples. This effect brings about the accuracy improvement of test samples. We demonstrated the effectiveness of the proposed target encoding through simulations of multi-layer perceptrons and convolutional neural networks for various classification problems such as handwritten-digit recognition, isolated-word recognition, image classification, and object recognition tasks. The simulation results show that the proposed probabilistic target encoding is superior to one-hot encoding as it prevents the overfitting of neural networks to training samples.
참고문헌 (24)
- S. -H. Oh, "Improving the Error Backpropagation Algorithm with a Modified Error Function", IEEE Trans. Neural Networks, 8(3), pp.799~803 (1997)[DOI]
- P. Rodriguez, "Beyond One-Hot Encoding: Lower Dimensional Target Encoding", Image and Vision Computing, 75, pp.21~31 (2018)[DOI]
- Z. Akata, "Label-Embedding for Image Classification", IEEE Trans. Pattern Analysis and Machine Intelligence, 38(7), pp.1425~1438 (2016)[DOI]
- S. Bengio, "Label Embedding Trees for Large Multi-Class Tasks", Advances in Neural Information Processing, pp.163~171 (2010)
- S. Escalera, "On the Decoding Process in Ternary Error-Correcting Output Codes", IEEE Trans. Pattern Analysis and Machine Intelligence, 32(1), pp.120~134 (2010)[DOI]
- H. White, "Learning in Artificial Neural Networks: a Statistical Perspective", Neural Computation, 1(4), pp.425~464 (1989)[DOI]
- M. D. Richard, "Neural Network Classifiers EstomateBayesian a Posteriori Probabilities", Neural Computation, 3(4), pp.461~483 (1991)[DOI]
- 오상훈, "A Statistical Perspective of Neural Networks for Imbalanced Data Problems", International Journal of Contents, 7(3), pp.1~5 (2011), 한국콘텐츠학회
- S. -H. Oh, "Statistical Analyses of Various Error Functions for Pattern Classifiers", Proc. Convergence on Hybrid Information Technology, Daejon, Korea, 206, pp.129~133 (2011)
- J. B. Hamshire II, "Equivalence Proofs for Multilayer Perceptron Classifier and the Bayesian Discriminant Function", Proc. 1990 Connectionist Model Summer School, pp.159~172 (1990)
- S. Horiguchi, "Significance of Softmax-Based Features in Comparison to Distance Metric Learning-Based Features", IEEE Trans. Pattern Analysis and Machine Intelligence, 42(5), pp.1279~1285 (2020)[DOI]
- J. B. Hampshire II, "A Novel Objective Function for Improved Phoneme Recognition Using Time-Delay Neural Networks", IEEE Trans. Neural Networks, 1, pp.216~228 (1990)[DOI]
- S. -H. Oh, "A Modified Error Function to Improve the Error Backpropagation Algorithm for Multilayer Perceptrons", ETRI Journal, 17(1), pp.11~22 (1995)[DOI]
- K. Baldi, "The Dropout Learning Algorithm", Artificial Intelligence, 210, pp.78~122 (2014)[DOI]
- N. Srivastava, "Dropout: A Simple Way to Prevent Neural Networks from Overfitting", Journal of Machine Learning researches, 15, pp.1929~1958 (2014)
- G. Gnecco, "The Weight-Decay Technique in Learning from Data: An Optimization Point of View", Computational Management Science, 6, pp.53~79 (2009)[DOI]
- J. J. Hull, "A Database for Handwritten Text Recognition Research", IEEE Trans. Pattern Anal. Machine Intell, 16(5), pp.550~554 (1994)[DOI]
- D. E. Rumelhart, "Parallel Distributed Processing" (1986), MIT Press
- D. -S. Kim, "Auditory Processing of Speech Signals for Robust Speech Recognition in Real-World Noisy Environment", IEEE Trans. Speech Audio Processing, 7(1), pp.55~69 (1999)[DOI]
- D. Kingma, "Adam: A Method for Stochastic Optimization", ~ (2014)
- L. Deng, "The MNIST database of Handwritten Digit Images for Machine Learning Research", IEEE Signal Processing Magazine, 29(6), pp.141~142 (2012)[DOI]
- "LeNet-5, Convoltuional Neural Networks"[URL]
- H. Xiao, "Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms", ~ (2017)
- Y. Abouelnaga, "CIFAR-10: KNN-based Ensemble of Classifiers", Int. Conf. Computational Science and Computational Intelligence, pp.1192~1195 (2016)