Impact of the Choice of Cross-Validation Techniques on the Results of Machine Learning-Based Diagnostic Applications
2
조회
0
다운로드
5
인용
0
참고문헌
키워드
초록
Objectives: With advances in data availability and computing capabilities, artificial intelligence and machine learning technologies have evolved rapidly in recent years. Researchers have taken advantage of these developments in healthcare informatics and created reliable tools to predict or classify diseases using machine learning-based algorithms. To correctly quantify the performance of those algorithms, the standard approach is to use cross-validation, where the algorithm is trained on a training set, and its performance is measured on a validation set. Both datasets should be subject-independent to simulate the expected behavior of a clinical study. This study compares two cross-validation strategies, the subject-wise and the record-wise techniques; the subject-wise strategy correctly mimics the process of a clinical study, while the record-wise strategy does not. Methods: We started by creating a dataset of smartphone audio recordings of subjects diagnosed with and without Parkinson’s disease. This dataset was then divided into training and holdout sets using subject-wise and the record-wise divisions. The training set was used to measure the performance of two classifiers (support vector machine and random forest) to compare six cross-validation techniques that simulated either the subject-wise process or the record-wise process. The holdout set was used to calculate the true error of the classifiers. Results: The record-wise division and the record-wise cross-validation techniques overestimated the performance of the classifiers and underestimated the classification error. Conclusions: In a diagnostic scenario, the subject-wise technique is the proper way of estimating a model’s performance, and record-wise techniques should be avoided.
Abstract
Objectives: With advances in data availability and computing capabilities, artificial intelligence and machine learning technologies have evolved rapidly in recent years. Researchers have taken advantage of these developments in healthcare informatics and created reliable tools to predict or classify diseases using machine learning-based algorithms. To correctly quantify the performance of those algorithms, the standard approach is to use cross-validation, where the algorithm is trained on a training set, and its performance is measured on a validation set. Both datasets should be subject-independent to simulate the expected behavior of a clinical study. This study compares two cross-validation strategies, the subject-wise and the record-wise techniques; the subject-wise strategy correctly mimics the process of a clinical study, while the record-wise strategy does not. Methods: We started by creating a dataset of smartphone audio recordings of subjects diagnosed with and without Parkinson’s disease. This dataset was then divided into training and holdout sets using subject-wise and the record-wise divisions. The training set was used to measure the performance of two classifiers (support vector machine and random forest) to compare six cross-validation techniques that simulated either the subject-wise process or the record-wise process. The holdout set was used to calculate the true error of the classifiers. Results: The record-wise division and the record-wise cross-validation techniques overestimated the performance of the classifiers and underestimated the classification error. Conclusions: In a diagnostic scenario, the subject-wise technique is the proper way of estimating a model’s performance, and record-wise techniques should be avoided.
참고문헌 (18)
- Perry B, "Use of mobile devices to measure outcomes in clinical research, 2010-2016 : a systematic literature review", Digit Biomark, 2(1), pp.11~30 (2018)
- Arlot S, "A survey of cross-validation procedures for model selection", Stat Surv, 4, pp.40~79 (2010)
- Bot BM, "The mPower study, Parkinson disease mobile data collected using ResearchKit", Sci Data, 3, pp.160011~ (2016)
- "The mPower Public Researcher Portal" (2019)[URL]
- "Synapse REST API: Basics Table Query" (2016)[URL]
- Rose S, "Why match? Investigating matched case-control study designs with causal effect estimation", Int J Biostat, 5(1), pp.1~ (2009)
- Wong SL, "Parkinson's disease : prevalence, diagnosis and impact", Health Rep, 25(11), pp.10~14 (2014)
- Khalid S, "A survey of feature selection and feature extraction techniques in machine learning", Proceedings of 2014 Science and Information Conference, pp.372~378 (2014)
- Giannakopoulos T, "Introduction to audio analysis: a MATLAB approach" (2014), Academic Press
- Giannakopoulos T, "pyAudioAnalysis : an open-source python library for audio signal analysis", PLoS One, 10(12), pp.e0144610~ (2015)
- Hawkins DM, "Assessing model fit by cross-validation", J Chem Inf Comput Sci, 43(2), pp.579~586 (2003)
- Pedregosa F, "Scikit-learn : machine learning in Python", J Mach Learn Res, 12, pp.2825~2830 (2011)
- "Stratified GroupKFold" (2019)[URL]
- Kotsiantis SB, "Data preprocessing for supervised leaning", Int J Comput Sci, 1(2), pp.111~117 (2006)
- Chandrashekar G, "A survey on feature selection methods", Comput Electr Eng, 40(1), pp.16~28 (2014)
- Zou H, "Regularization and variable selection via the elastic net", J R Stat Soc Series B Stat Methodol, 67(2), pp.301~320 (2005)
- Wager S, "Cross-validation, risk estimation, and model selection : comment on a paper by Rosset and Tibshirani", J Am Stat Assoc, 115(529), pp.157~160 (2020)
- Saeb S, "The need to approximate the use-case in clinical machine learning", Gigascience, 6(5), pp.1~9 (2017)