Statistical Analyses of Cross-Entropy Error Function with Probabilistic Target Encoding for Training Neural Networks

Abstract

Training neural networks with softmax outputs requires assigning target values to output nodes. Due to its simplicity, we often use one-hot encoding, which adopts “one” or “zero” as the desired output values. However, when training neural networks to minimize the cross-entropy error function between the desired and actual output node values, overfitting of neural networks to training samples becomes a significant issue. A probabilistic target encoding has been proposed to mitigate the overfitting. In this paper, we derive the optimal solutions for output nodes that minimize the cross-entropy error function using the probabilistic target encoding. In the extreme case of the probabilistic target encoding, the analysis corresponds to the cross-entropy error function with one-hot encoding. The statistical analyses conducted to derive the optimal solutions provide considerable insights, including the interval of target values for the Bayes classifier.