본문 바로가기
KERD
논문KCI 등재

Comparative Study of the Performance of Support Vector Machines with Various Kernels

김현민|김상일|유용빈|남성욱
영남수학회 (Youngnam Mathematical Society)|2021.05|East Asian Mathematical Journal|제37권 제3호|pp. 333-354

1

조회

0

다운로드

0

인용

0

참고문헌

키워드

Machine LearningSupport Vector MachineKernel FunctionsTaylor Diagram
주제분류수학

초록

A support vector machine (SVM) is a state-of-the-art machine learning model rooted in structural risk minimization. SVM is underestimated with regards to its application to real world problems because of the difficulties associated with its use. We aim at showing that the performance of SVM highly depends on which kernel function to use. To achieve these, after providing a summary of support vector machines and kernel function, we constructed experiments with various benchmark datasets to compare the performance of various kernel functions. For evaluating the performance of SVM, the F1-score and its Standard Deviation with 10-cross validation was used. Furthermore, we used taylor diagrams to reveal the difference between kernels. Finally, we provided Python codes for all our experiments to enable re-implementation of the experiments.

Abstract

A support vector machine (SVM) is a state-of-the-art machine learning model rooted in structural risk minimization. SVM is underestimated with regards to its application to real world problems because of the difficulties associated with its use. We aim at showing that the performance of SVM highly depends on which kernel function to use. To achieve these, after providing a summary of support vector machines and kernel function, we constructed experiments with various benchmark datasets to compare the performance of various kernel functions. For evaluating the performance of SVM, the F1-score and its Standard Deviation with 10-cross validation was used. Furthermore, we used taylor diagrams to reveal the difference between kernels. Finally, we provided Python codes for all our experiments to enable re-implementation of the experiments.

참고문헌 (95)

  1. Boser, Bernhard E, "A training algorithm for optimal margin classifiers", Proceedings of the fifth annual workshop on Computational learning theory, pp.144~152 (1992), ACM
  2. Smola, "A tutorial on support vector regression", Statistics and computing, 14(3), pp.199~222 (2004), Springer
  3. Burges, Christopher JC, "A tutorial on support vector machines for pattern recognition", Data mining and knowledge discovery, 2(2), pp.121~167 (1998), Springer
  4. "Svm application list" (1999)[URL]
  5. Wang, Guosheng, "A survey on training algorithms for support vector machine classifiers", Fourth International Conference on Networked Computing and Advanced Information Management, 1, pp.123~128 (2008), IEEE
  6. Souza, Cesar R, "Kernel functions for machine learning applications", Creative Commons Attribution-Noncommercial-Share Alike, 3, pp.29~ (2010)
  7. Shawe-Taylor, John, "A review of optimization methodologies in support vector machines", Neurocomputing, 74(17), pp.3609~3618 (2011), Elsevier
  8. Nayak, Janmenjoy, "A comprehensive survey on sup-port vector machine in data mining tasks: applications & challenges", International Journal of Database Theory and Application, 8(1), pp.169~186 (2015)
  9. Campbell, Colin, "Kernel methods: a survey of current techniques", Neurocomputing, 48(1-4), pp.63~84 (2002), Elsevier
  10. Smola, Alex J, "Learning with kernels, 4", Citeseer, ~ (1998)
  11. Vedaldi, Andrea, "Efficient additive kernels via explicit feature maps", IEEE transactions on pattern analysis and machine intelligence, 34(3), pp.480~492 (2012), IEEE
  12. Rahimi, Ali, "Random features for large-scale kernel machines", Advances in neural information processing systems, pp.1177~1184 (2008)
  13. Mangasarian, Olvi L, "Cancer diagnosis via linear programming", University of Wisconsin-Madison Department of Computer Sciences, ~ (1990)
  14. UCI Machine Learning Repository, "Dua, Dheeru", University of Califor-nia, Irvine, School of Information and Computer Sciences, ~ (2017)
  15. Silva, Pedro FB, "Evaluation of features for leaf discrimination", International Conference Image Analysis and Recognition, pp.197~204 (2013), Springer
  16. Redmond, Michael, "A data-driven software tool for enabling cooperative information sharing among police departments", European Journal of Operational Research, 141(3), pp.660~678 (2002), Elsevier
  17. Brooks, Thomas F, "Airfoil self-noise and prediction", ~ (1989)
  18. Cortez, Paulo, "A data mining approach to pre-dict forest fires using meteorological data, Data mining and knowledge discovery", APPIA, ~ (2007)
  19. Cassotti, M, "A similarity-based QSAR model for predicting acute toxicity towards the fathead minnow (Pimephales promelas)", SAR and QSAR in Environmental Research, 26(3), pp.217~243 (2015), Taylor & Francis
  20. Nakai, Kenta, "Expert system for predicting protein localization sites in gram-negative bacteria", Proteins: Structure, Function, and Bioinformatics, 11(2), pp.95~110 (1991), Springer
  21. Breiman, Leo, "Classification and regression trees" (2017), Routledge
  22. Cortez, Paulo, "Modeling wine preferences by data mining from physicochemical properties", Decision Support Systems, 47(4), pp.547~553 (1998), Elsevier
  23. Taylor, Karl E, "Summarizing multiple aspects of model performance in a single diagram", Journal of Geophysical Research: Atmospheres, 106(D7), pp.7183~7192 (1998)
  24. Pedregosa, Fabian, "Scikit-learn: Machine learning in Python", Journal of machine learning research, 12, pp.2825~2830 (2011)
  25. McKinney, Wes, "Data structures for statistical computing in python", Proceed-ings of the 9th Python in Science Conference, 445, pp.51~56 (2010)
  26. "taylor diagram python code" (2018)[URL]
  27. Support vector machine-reference manual, "Saunders, Craig", Department of Computer Science, Royal Holloway, University of London, ~ (1998)
  28. Valentini, Giorgio, "Gene expression data analysis of human lymphoma using support vec-tor machines and output coding ensembles", Artificial Intelligence in Medicine, 26(3), pp.281~304 (2002), Elsevier
  29. Fadel, Sayed, "Investigating the effect of different kernel functions on the performance of SVM for recognizing Arabic characters", International Journal of Advanced Computer Science and Applications, 7(1), pp.446~450 (2016)
  30. Chen, Rung-Ching, "Web page classification based on a support vector machine using a weighted vote schema", Expert Systems with Applications, 31(2), pp.427~435 (2006), Elsevier
  31. Kar, Purushottam, "Random feature maps for dot product kernels", Artificial Intelligence and Statistics, pp.583~591 (2012)
  32. Deng, Wan-Yu, "A fast reduced kernel extreme learning machine", Neural Networks, 76, pp.29~38 (2016), Elsevier
  33. Wang, Benjamin X, "Boosting support vector machines for im-balanced data sets", Knowledge and information systems, 25(1), pp.1~20 (2010), Springer
  34. Caruana, Rich, "Alexandru, An empirical comparison of supervised learning algorithms", Proceedings of the 23rd international conference on Machine learning, pp.161~168 (2006), ACM
  35. Alashwal, Hany, "A Bayesian kernel for the Prediction of Protein-Protein Interactions, World Academy of Science", Engineering and Technology, 51, pp.928~933 (2009)
  36. BONITA, OLIVIA, "Comparison of Gaussian and ANOVA Kernel in Support Vector Regression for Predicting Coal Price", 2018 International Con-ference on Sustainable Information Engineering and Technology (SIET), pp.147~150 (2018), IEEE
  37. Gish, Herbert, "A probabilistic approach to the understanding and training of neural network classifiers", International Conference on Acoustics, Speech, and Signal Processing, pp.1361~1364 (1990), IEEE
  38. Zhang, Guoqiang Peter, "Guoqiang Peter, Neural networks for classification: a survey", IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 30(4), pp.451~462 (2000), IEEE
  39. Adya, Monica, "How effective are neural networks at forecasting and prediction? A review and evaluation", Journal of forecasting, 17(5-6), pp.481~495 (1998), Wiley Online Library
  40. Callen, Jeffrey L, "Neural network forecasting of quarterly accounting earnings", International Journal of Forecasting, 12(4), pp.475~482 (1996), Elsevier
  41. Church, Keith B, "Forecasting consumers’ expenditure: A compar-ison between econometric and neural network models", International journal of forecasting, 12(2), pp.255~267 (1996), Elsevier
  42. Connor, Jerome T, "Recurrent neural networks and robust time series prediction", IEEE transactions on neural networks, 5(2), pp.240~254 (1994), IEEE
  43. Cottrell, Marie, "Neural modeling for time series: a statistical stepwise method for weight elimination", IEEE transactions on neural networks, 6(6), pp.1355~1364, IEEE
  44. Faraway, Julian, "Time series forecasting with neural networks: a comparative study using the air line data", Journal of the Royal Statistical Society: Series C (Applied Statistics), 47(2), pp.231~250 (1998), Wiley Online Library
  45. Fletcher, Desmond, "Forecasting with neural networks: an application using bankruptcy data", Information & Management, 24(3), pp.159~167 (1993), Elsevier
  46. Gorr, Wilpen L, "Research prospective on neural network forecasting", International Jour-nal of Forecasting, 10(1), pp.1~4 (1994), Elsevier
  47. Hippert, Henrique Steinherz, "Neural networks for short-term load forecasting: A review and evaluation", IEEE Trans-actions on power systems, 16(1), pp.44~55 (2001), IEEE
  48. Belli, MR, "Artificial neural networks as approximators of stochastic processes", Neural Networks, 12(4-5), pp.647~658 (1999), Elsevier
  49. Castro, Juan Luis, "Neural networks with a continuous squashing function in the output are universal approximators", Neural Networks, 13(6), pp.561~563 (2000), Elsevier
  50. Funahashi, Ken-Ichi, "On the approximate realization of continuous mappings by neural networks", Neural networks, 2(3), pp.183~192 (1989), Elsevier
  51. Andrews, Robert, "Survey and critique of techniques for extracting rules from trained artificial neural networks", Knowledge-based systems, 8(6), pp.373~389 (1995), Elsevier
  52. Castro, Juan L, "Interpretation of artifi-cial neural networks by means of fuzzy rules", IEEE Transactions on Neural Networks, 13(1), pp.101~116 (2002), IEEE
  53. Setiono, Rudy, "Extraction of rules from artificial neural networks for nonlinear regression", IEEE transactions on neural networks, 13(3), pp.564~577 (2002), IEEE
  54. Setiono, Rudy, "An approach to generate rules from neural net-works for regression problems", European Journal of Operational Research, 155(1), pp.239~250 (2004), Elsevier
  55. Lisboa, Paulo JG, "A review of evidence of health benefit from artificial neural networks in medical intervention", Neural networks, 15(1), pp.11~39 (2002), Elsevier
  56. Portney, Leslie Gross, "Foundations of clinical research: applications to practice", pp.2009, Pearson
  57. Shawe-Taylor, John, "Structural risk minimization over data-dependent hierarchies", IEEE transactions on Information Theory, 44(5), pp.1926~1940 (1998), IEEE
  58. Vapnik, Vladimir, "Estimation of dependences based on empirical data" (2006), Springer Science & Business Media
  59. McCulloch, Warren S, "A logical calculus of the ideas immanent in nervous activity", The bulletin of mathematical biophysics, 5(4), pp.115~133 (1943), Springer
  60. McClelland, James L, "Parallel distributed processing", pp.2~ (1987), MIT press
  61. Dietterich, Thomas G, "Ensemble methods in machine learning", International workshop on multiple classifier systems, pp.1~15 (2000), Springer
  62. Rokach, Lior, "Feature set decomposition for decision trees", Intelligent Data Analysis, 9(2), pp.131~158 (1998), IOS Press
  63. Kuncheva, Ludmila I, "Measures of diversity in classifier en-sembles and their relationship with the ensemble accuracy", Machine learning, 51(2), pp.181~207 (2003), Springer
  64. Sollich, Peter, "Learning with ensembles: How overfitting can be useful", Advances in neural information processing systems, pp.190~196 (1996)
  65. Brown, Gavin, "Diversity creation methods: a survey and categorisation", Information Fusion, 6(1), pp.5~20 (2005), Elsevier
  66. Adeva, Juan, "Accuracy and diversity in ensembles of text categorisers", Electronic Journal, 9(1), pp.1~12 (2005)
  67. Krogh, Anders, "Neural network ensembles, cross validation, and active learning", Advances in neural information processing systems, pp.231~238 (1995)
  68. Belkin, Mikhail, "Reconciling modern machine-learning practice and the classical bias–variance trade-off", Proceedings of the National Academy of Sciences, 116(32), pp.15849~15854, National Acad Sciences
  69. Nakkiran, Preetum, "Deep double descent: Where bigger models and more data hurt", arXiv preprint arXiv:1912.02292, ~ (2019)
  70. Cybenko, George, "Approximation by superpositions of a sigmoidal function", Mathematics of control, signals and systems, 2(4), pp.303~314 (1989), Springer
  71. Joachims, Thorsten, "Training linear SVMs in linear time", Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp.217~226 (2006), ACM
  72. Boughorbel, Sabri, "Conditionally positive definite kernels for svm based image recognition", 2005 IEEE International Conference on Multi-media and Expo, pp.113~116 (2005), IEEE
  73. Nasrabadi, Nasser M, "Kernel spectral matched filter for hyperspectral target detection", Proceedings.(ICASSP’05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 4, pp.4~665 (2005), IEEE
  74. Boughorbel, Sabri, "The GCS kernel for SVM-based image recognition", International Conference on Artificial Neural Networks, pp.595~600 (2005), Springer
  75. Chiroma, Haruna, "Kernel functions for the support vector machine: comparing performances on crude oil price data", Recent Advances on Soft Computing and Data Mining, pp.273~281 (2014), Springer
  76. Fleuret, Francois, "Scale-invariance of support vector machines based on the triangular kernel", 3rd International Workshop on Statistical and Computational Theories of Vision, pp.1~13 (2003)
  77. Achirul Nanda, Muhammad, "A comparison study of kernel functions in the support vector machine and its application for termite detection", Information, 9(1) (2018), Multidisciplinary Digital Publishing Institute
  78. Gunn, Steve R, "Support vector machines for classification and regression", ISIS technical report, 14(1), pp.5~16 (1998), University of Southampton
  79. Maji, Subhransu, "Efficient classification for additive kernel SVMs", IEEE transactions on pattern analysis and machine intelligence, 35(1), pp.66~77 (2012)
  80. Vanek, Jan, "A Comparison of Support Vec-tor Machines Training GPU-Accelerated Open Source Implementations", arXiv preprint arXiv:1707.06470, ~ (2017)
  81. Afifi, Shereen Moataz, "Hardware implemen-tations of SVM on FPGA: A state-of-the-art review of current practice", International Journal of Innovative Science Engineering and Technology (IJISET) (2015)
  82. Burges, Christopher JC, "Speed up SVM algorithm for massive classification tasks", Inter-national conference on advanced data mining and applications, pp.2008~2147 (2008), Springer
  83. A new method for constructing artificial neural networks, Interim technical report, "Burges, CJC", ONR contract, ~ (1995)
  84. Abiodun, Oludare Isaac, "State-of-the-art in artificial neural network applications: A survey", Heliyon, 4(11) (2018), Elsevier
  85. Saravanan, Kl, "Review on classification based on artificial neural net-works", International Journal of Ambient Systems and Applications (IJASA), 2(4), pp.11~18 (2014)
  86. Martinez-Porchas, Marcel, "Enrique and Vargas-Albores, Fran-cisco, Significant loss of sensitivity and specificity in the taxonomic classification occurs when short 16S rRNA gene sequences are used", Heliyon, 2(9) (2016), Elsevier
  87. Abid, Faroudja, "A survey of neural network based automated systems for human chromosome classification", Artificial Intelligence Review, 49(1), pp.41~56 (2018), Springer
  88. Wang, Yaohui, "Application of SVM in object tracking based on Lapla-cian kernel function", 8th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC) 2, pp.557~561 (2016), IEEE
  89. Zhang, Li, "Wavelet support vector machine", IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 34(1), pp.34~39 (2004), IEEE
  90. Xiang, Li, "Research of bessel kernel function of the first kind for support vector regression", Information Technology Journal, 12(14), pp.2673~2682 (2013), ANSINET
  91. Horvath, Gabor, "CMAC neural network as an SVM with B-Spline kernel functions", Pro-ceedings of the 20th IEEE Instrumentation Technology Conference, 2, pp.1108~1113 (2003), IEEE
  92. Aftab, Wasim, "A novel kernel for RBF based neural networks" (2014), Abstract and Applied Analysis
  93. Abadi, Wassila, "Bag of Visualwords and Chi-Squared Kernel Support Vector Machine: A Way to Improve Hand Gesture Recognition", Proceedings of the International Conference on Intelligent Information Processing, Secu-rity and Advanced Communication (2015), ACM
  94. Rao, Swathi, "Effects of Image Retrieval from Image Database using Linear Kernel and Hellinger Kernel Mapping of SVM", International Journal of Scientific & Engineering Re-search, 4(5)
  95. Roul, "A Modified Cosine-Similarity based Log Kernel for Support Vector Machines in the Domain of Text Classification", Proceedings of the 14th International Conference on Natural Language Processing, pp.338~347 (2017)

관련 논문AI추천

Multi-output Kernel Regression for Correlated Multiple Outputs박혜정, 석경하, 심주용자연과학연구소 · Quantitative Bio-Science2021 · 제40권 제2호Re-SSS: Rebalancing Imbalanced Data Using Safe Sample ScreeningHongbo Shi, Min Guo, Xin Chen한국정보처리학회 · JIPS(Journal of Information Processing Systems)2021 · 제17권 제1호Study of Personal Credit Risk Assessment Based on SVMLI Xin, XIA Han한국유통과학회 · 산경연구논집2022 · 제13권 제10호Power Quality Disturbances Identification Method Based on Novel Hybrid Kernel FunctionLiquan Zhao, Meijiao Gai한국정보처리학회 · JIPS(Journal of Information Processing Systems)2019 · 제15권 제2호Default Prediction of Automobile Credit Based on Support Vector MachineRuirui Zhang, Ying Chen한국정보처리학회 · JIPS(Journal of Information Processing Systems)2021 · 제17권 제1호COMPARATIVE ANALYSIS ON MACHINE LEARNING MODELS FOR PREDICTING KOSPI200 INDEX RETURNS구본상, 송준혁한국수학교육학회 · 이론수학과 교직수학2017 · 제24권 제4호

이 간행물의 다른 논문 (4건)

Degree of Approximation of Functions of Class $Zyg^{\omega}(\alpha,\gamma)$ by Cesaro means of Fourier series김재만pp. 289-293Efficiency Analysis of a Domain Decomposition Method for the Two-dimensional Telegraph Equations전윤배pp. 295-305An Intrinsic Criterion for the Type of Automorphisms of the Unit Disc이강혁pp. 307-317Certain New Families for Bi-univalent Functions Defined by a Known Operator최준상, Abbas Kareem Wanaspp. 319-331
원문 보기