The clinical meaning of the area under a receiver operating characteristic curve for the evaluation of the performance of disease markers
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
The area under a receiver operating characteristic (ROC) curve (AUC) is a popular measure of pure diagnostic accuracy that is independent from the proportion of diseased subjects in the analysed sample. However, its actual usefulness in the clinical context has been questioned, because it does not seem to be directly related to the actual performance of a diagnostic marker in identifying diseased and non-diseased subjects in real clinical settings. This study evaluates the relationship between the AUC and the proportion of correct classifications (global diagnostic accuracy, GDA) in relation to the shape of the corresponding ROC curves. We demonstrate that AUC represents an upward-biased measure of GDA at an optimal accuracy cut-off for balanced groups. The magnitude of bias depends on the shape of the ROC plot and on the proportion of diseased and non-diseased subjects. In proper curves, the bias is independent from the diseased/non-diseased ratio and can be easily estimated and removed. Moreover, a comparison between 2 partial AUCs can be replaced by a more powerful test for the corresponding whole AUCs. Applications to 3 real datasets are provided: a marker for a hormone deficit in children, 2 tumour markers for malignant mesothelioma, and 2 gene expression profiles in ovarian cancer patients. In conclusion, the AUC is a measure of accuracy with potential clinical relevance for the evaluation of disease markers. The clinical meaning of ROC parameters should always be evaluated with an analysis of the shape of the corresponding ROC curve.
Abstract
The area under a receiver operating characteristic (ROC) curve (AUC) is a popular measure of pure diagnostic accuracy that is independent from the proportion of diseased subjects in the analysed sample. However, its actual usefulness in the clinical context has been questioned, because it does not seem to be directly related to the actual performance of a diagnostic marker in identifying diseased and non-diseased subjects in real clinical settings. This study evaluates the relationship between the AUC and the proportion of correct classifications (global diagnostic accuracy, GDA) in relation to the shape of the corresponding ROC curves. We demonstrate that AUC represents an upward-biased measure of GDA at an optimal accuracy cut-off for balanced groups. The magnitude of bias depends on the shape of the ROC plot and on the proportion of diseased and non-diseased subjects. In proper curves, the bias is independent from the diseased/non-diseased ratio and can be easily estimated and removed. Moreover, a comparison between 2 partial AUCs can be replaced by a more powerful test for the corresponding whole AUCs. Applications to 3 real datasets are provided: a marker for a hormone deficit in children, 2 tumour markers for malignant mesothelioma, and 2 gene expression profiles in ovarian cancer patients. In conclusion, the AUC is a measure of accuracy with potential clinical relevance for the evaluation of disease markers. The clinical meaning of ROC parameters should always be evaluated with an analysis of the shape of the corresponding ROC curve.
참고문헌 (35)
- Bamber D, "The area above the ordinal dominance graph and the area below the receiver operating characteristic graph", J Math Psychol, 12, pp.387~415 (1975)
- Janes H, "Accommodating covariates in ROC analysis", Stata J, 9, pp.17~39 (2009)
- Pepe M, "Estimation and comparison of receiver operating characteristic curves", Stata J, 9, pp.1~ (2009)
- Vickers AJ, "Incorporating clinical considerations into statistical analyses of markers : a quiet revolution in how we think about data", Clin Chem, 62, pp.671~672 (2016)
- Janssens AC, "Reflection on modern methods : revisiting the area under the ROC curve", Int J Epidemiol, 49, pp.1397~1403 (2020)
- McClish DK, "Analyzing a portion of the ROC curve", Med Decis Making, 9, pp.190~195 (1989)
- Jiang Y, "A receiver operating characteristic partial area index for highly sensitive diagnostic tests", Radiology, 201, pp.745~750 (1996)
- Pepe MS, "The statistical evaluation of medical tests for classification and prediction" (2003), Oxford University Press
- Robin X, "pROC : an open-source package for R and S+ to analyze and compare ROC curves", BMC Bioinformatics, 12, pp.77~ (2011)
- Bandos AI, "Jackknife variance of the partial area under the empirical receiver operating characteristic curve", Stat Methods Med Res, 26, pp.528~541 (2017)
- Ma H, "On the use of partial area under the ROC curve for comparison of two diagnostic tests", Biom J, 57, pp.304~320 (2015)
- "R: a language and environment for statistical computing" (2015)[URL]
- Pepe MS, "Phases of biomarker development for early detection of cancer", J Natl Cancer Inst, 93, pp.1054~1061 (2001)
- Eusebi P, "Diagnostic accuracy measures", Cerebrovasc Dis, 36, pp.267~272 (2013)
- Pepe MS, "Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker", Am J Epidemiol, 159, pp.882~890 (2004)
- Bandos AI, "Estimating the area under ROC curve when the fitted binormal curves demonstrate improper shape", Acad Radiol, 24, pp.209~219 (2017)
- Metz CE, "“Proper” binormal ROC curves: theory and maximum-likelihood estimation", J Math Psychol, 43, pp.1~33 (1999)
- Metz CE, "Statistical significance tests for binormal ROC curves", J Math Psychol, 22, pp.218~243 (1980)
- Dorfman DD, "Proper receiver operating characteristic analysis : the bigamma model", Acad Radiol, 4, pp.138~149 (1997)
- Walter SD, "Properties of the summary receiver operating characteristic(SROC)curve for diagnostic test data", Stat Med, 21, pp.1237~1256 (2002)
- DeLong ER, "Comparing the areas under two or more correlated receiver operating characteristic curves : a nonparametric approach", Biometrics, 44, pp.837~845 (1988)
- Walter SD, "The partial area under the summary ROC curve", Stat Med, 24, pp.2025~2040 (2005)
- Secco A, "Reassessment of the growth hormone status in young adults with childhood-onset growth hormone deficiency : reappraisal of insulin tolerance testing", J Clin Endocrinol Metab, 94, pp.4195~4204 (2009)
- Filiberti R, "Diagnostic value of mesothelin in pleural fluids : comparison with CYFRA 21-1 and CEA", Med Oncol, 30, pp.543~ (2013)
- Erdreich LS, "Use of relative operating characteristic analysis in epidemiology. A method for dealing with subjective judgement", Am J Epidemiol, 114, pp.649~662 (1981)
- Baker SG, "Peirce, Youden, and receiver operating characteristic curves", Am Stat, 61, pp.343~346 (2007)
- Martínez-Camblor P, "The area under the generalized receiver-operating characteristic curve", Int J Biostat, 18, pp.293~306 (2021)
- Bantis LE, "The length of the receiver operating characteristic curve and the two cutoffYouden index within a robust framework for discovery, evaluation, and cutoffestimation in biomarker studies involving improper receiver operating characteristic curves", Stat Med, 40, pp.1767~1789 (2021)
- Kagaris D, "A multi-index ROC-based methodology for high throughput experiments in gene discovery", Int J Data Min Bioinform, 8, pp.42~65 (2013)
- Lee WC, "Alternative summary indices for the receiver operating characteristic curve", Epidemiology, 7, pp.605~611 (1996)
- Parodi S, "Not proper ROC curves as new tool for the analysis of differentially expressed genes in microarray experiments", BMC Bioinformatics, 9, pp.410~ (2008)
- Pepe MS, "Selecting differentially expressed genes from microarray experiments", Biometrics, 59, pp.133~142 (2003)
- Parodi S, "Restricted ROC curves are useful tools to evaluate the performance of tumour markers", Stat Methods Med Res, 25, pp.294~314 (2016)
- Cook NR, "Use and misuse of the receiver operating characteristic curve in risk prediction", Circulation, 115, pp.928~935 (2007)
- Pepe MS, "The net reclassification index(NRI) : a misleading measure of prediction improvement even with independent test data sets", Stat Biosci, 7, pp.282~295 (2015)