Deriving a New Divergence Measure from Extended Cross-Entropy Error Function
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Relative entropy is a divergence measure between two probability density functions of a random variable. Assuming that the random variable has only two alphabets, the relative entropy becomes a cross-entropy error function that can accelerate training convergence of multi-layer perceptron neural networks. Also, the n-th order extension of cross-entropy (nCE) error function exhibits an improved performance in viewpoints of learning convergence and generalization capability. In this paper, we derive a new divergence measure between two probability density functions from the nCE error function. And the new divergence measure is compared with the relative entropy through the use of three-dimensional plots.
Abstract
Relative entropy is a divergence measure between two probability density functions of a random variable. Assuming that the random variable has only two alphabets, the relative entropy becomes a cross-entropy error function that can accelerate training convergence of multi-layer perceptron neural networks. Also, the n-th order extension of cross-entropy (nCE) error function exhibits an improved performance in viewpoints of learning convergence and generalization capability. In this paper, we derive a new divergence measure between two probability density functions from the nCE error function. And the new divergence measure is compared with the relative entropy through the use of three-dimensional plots.
참고문헌 (18)
- K. Hornik, "Multilayer Feed-forward Networks are Universal Approximators", Neural Networks, 2, pp.359~366 (1989)
- K. Hornik, "Approximation Capabilities of Multilayer Feedforward Networks", Neural Networks, 4, pp.251~257 (1991)
- S. Suzuki, "Constructive Function Approximation by Three-Layer Artificial Neural Networks", Neural Networks, 11, pp.1049~1058 (1998)
- D. E. Rumelhart, "Parallel Distributed Processing" (1986), Cambridge
- A. van Ooyen, "Improving the Convergence of the Backpropagation Algorithm", Neural Networks, 5, pp.465~471 (1992)
- S.-H. Oh, "Improving the Error Back-Propagation Algorithm with a Modified Error Function", IEEE Trans. Neural Networks, 8, pp.799~803 (1997)
- A. El-Jaroudi, "A New Error Criterion for Posterior probability Estimation with Neural Nets", Proc. IJCNN’90, III, pp.185~192 (1990)
- M. Bichsel, "Minimum Class Entropy: A maximum Information Approach to Layered Networks", Neural Networks, 2, pp.133~141 (1989)
- S. Ridella, "Representation and Generalization Properties of Class-Entropy Networks", IEEE Trans. Neural Networks, 10, pp.31~47 (1999)
- D. Erdogmus, "Entropy Minimization Algorithm for Multilayer Perceptrons", Proc. IJCNN’01, 4, pp.3003~3008 (2001)
- K. E. Hild II, "Feature Extraction Using Information-Theoretic Learning", IEEE Trans. PAMI, 28(9), pp.1385~1392 (2006)
- S.-J. Lee, "Constructing Neural Networks for Multiclass-Discretization Based on Information Theory", IEEE Trans. Sys., Man, and Cyb.-Part B, 29, pp.445~453 (1999)
- D. Erdogmus, "Information Transfer Through Classifiers and Its Relation to Probability of Error", Proc. IJCNN’01, 1, pp.50~54 (2001)
- R. Kamimura, "Hidden Information maximization for Feature Detection and Rule Discovery", Computation in Neural Systems, 6, pp.577~602 (1995)
- K. Torkkola, "Nonlinear Feature Transforms Using Maximum Mutual Information", Proc. IJCNN’01, 4, pp.2756~2761 (2001)
- T. M. Cover, "Elements of Information Theory" (1991), John Wiley & Sons
- S.-H. Oh, "Contour Plots of Objective Functions for Feed-Forward Neural Networks", Int. Journal of Contents, 8(4), pp.30~35 (2012)
- S.-H. Oh, "Statistical Analyses of Various Error Functions For Pattern Classifiers", CCIS, 206, pp.129~133 (2011)