Concept Drift에 의한 ML 모델 성능 변화의 정량적 추정 방법
Quantitative Estimation Method for ML Model Performance Change, Due to Concept Drift
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
기계학습을 통해 학습된 모델은 업무 활용 시 그 성능을 실측하기 매우 어렵다. 때문에 운영 부서에서는 모델의 성능을 효과적으로 관리하지못한다. 이로 인해 모델의 상태를 판단하기 위한 Concept drift 탐지 방법이 다양하게 연구되고 있다. 운영 부서에서는 운영 중인 모델의 성능을정량적으로 관리하려고 한다. 그러나 Concept drift는 모델 상태를 데이터 관계적으로 판단 할 뿐, 모델의 정량적 성능 수치를 추정하지는 못한다. 본 연구에서는 Concept drift의 통계량을 통해 정량적으로 precision 값을 추정하는 성능 예측 모델(PPM, Performance prediction model)을제안한다. 제안 모델의 Algorithm 1에서는, 학습데이터에서 복원 추출한 샘플링 데이터에 인위적인 drift를 유도하고 이때의 precision을 측정하여drift와 precision의 데이터 셋을 만들어 학습한다. Algorithm 2에서는 테스트 데이터를 통해 실제 precision과 예측 precision의 차이를 측정하여성능 예측 모델의 오차를 보정 한다. 현실 비즈니스에서 사용될 수 있는 대출 심사 모델과 신용카드 오사용 탐지 모델에 PPM을 적용하여 성능예측의 유효성을 확인했다.
Abstract
It is very difficult to measure the performance of the machine learning model in the business service stage. Therefore, managing theperformance of the model through the operational department is not done effectively. Academically, various studies have been conductedon the concept drift detection method to determine whether the model status is appropriate. The operational department wants to knowquantitatively the performance of the operating model, but concept drift can only detect the state of the model in relation to the data,it cannot estimate the quantitative performance of the model. In this study, we propose a performance prediction model (PPM) thatquantitatively estimates precision through the statistics of concept drift. The proposed model induces artificial drift in the sampling dataextracted from the training data, measures the precision of the sampling data, creates a dataset of drift and precision, and learns it. Then, the difference between the actual precision and the predicted precision is compared through the test data to correct the errorof the performance prediction model. The proposed PPM was applied to two models, a loan underwriting model and a credit card frauddetection model that can be used in real business. It was confirmed that the precision was effectively predicted.
참고문헌 (34)
- BHOWMIK, Pritom, "Machine Learning in Production:From Experimented ML Model to System", ~ (2022)
- "Machine Learning Monitoring"[URL]
- "Data Drift vs. Concept Drift"[URL]
- M. Salganicoff, "Tolerating concept and sampling shift in lazy learning using prediction error context switching", Artificial Intelligence Review, 11(1-5), pp.133~155 (1997)
- S. Rabanser, "Failing loudly : An empirical study of methods for detecting dataset shift", Advances in Neural Information Processing Systems, 32 (2019)
- L. I. Kuncheva, "Classifier ensembles for changing environments", Multiple Classifier Systems: 5th International Workshop, MCS 2004, pp.1~15 (2004), Springer Berlin Heidelberg
- L. Kuncheva, "Classifier ensembles for detecting concept change in streaming data : Overview and perspectives", Proceedings of the 2nd Workshop SUEMA 2008 (2008)
- J. Gama, "Learning with drift detection, in Advances in Artificial Intelligence:SBIA 2004", LNCS, 3171, pp.286~295 (2004), Springer
- M. Baena-Garcia, "Early drift detection method", International Workshop on Knowledge Discovery from Data Streams, pp.77~86 (2006)
- A. Bifet, "Learning from time-changing data with adaptive windowing", Proceedings of the 7th SIAM International Conference on Data Mining (SDM’07), pp.443~448 (2007)
- R. Pears, "Detecting concept change in dynamic data streams", Machine Learning, 97(3), pp.259~293 (2014)
- S. Bernstein, "The theory of probabilities" (1946), Gastehizdat Publishing House
- D. T. J. Huang, "Detecting volatility shift in data streams", Proceedings of 2014 IEEE International Conference on Data Mining ICDM, pp.863~868 (2014)
- K. Nishida, "Detecting concept drift using statistical testing, in Proceedings of 10th International Conference on Discovery Science (DS’07)", LNCS, 4755, pp.264~269 (2007), Springer
- R. S. M. Barros, "Advances in data stream mining with concept drift", pp.2017
- R. S. M. Barros, "Wilcoxon rank sum test drift detector", Neurocomputing, 275, pp.1954~1963 (2018)
- D. R. L. Cabral, "Statistical tests and detection of concept drifts in data streams", M.Sc, pp.2017
- D. R. L. Cabral, "Concept drift detection based on fisher’s exact test", Information Sciences, 442, pp.220~234 (2018)
- R. Fisher, "On the interpretation of χ2from contingency tables, and the calculation of P", Journal of the Royal Statistical Society, 85(1), pp.87~94 (1922)
- J. Z. Kolter, "Dynamic weighted majority : an ensemble method for drifting concepts", The Journal of Machine Learning Research, 8, pp.2755~2790 (2007)
- A. Blum, "Empirical support for winnow and weightedmajority algorithms : Results on a calendar scheduling domain", Machine Learning, 26(1), pp.5~23 (1997)
- L. Breiman, "Bagging predictors", Machine Learning, 24(2), pp.123~140 (1996)
- Y. Freund, "Experiments with a new boosting algorithm", International Conference on Machine Learning, 96, pp.148~156 (1996)
- N. C. Oza, "Artificial Intelligence and Statistics", pp.105~112 (2001), Morgan Kaufman
- A. Bifet, "New ensemble methods for evolving data streams", Proceedings of the 15th ACM International Conference on Knowledge Discovery and Data Mining (KDD’09), pp.139~148 (2009)
- A. Bifet, "Leveraging bagging for evolving data streams, in Machine Learning and Knowledge Discovery in Databases", LNCS, 6321, pp.135~150 (2010), Springer
- L. L. Minku, "DDD : a new ensemble approach for dealing with concept drift", IEEE transactions on Knowledge and Data Engineering, 24(4), pp.619~633 (2012)
- I. Frías-Blanco, "Fast adaptive stack-ing of ensembles", Proceedings of the 31st ACM Symposium on Applied Computing (SAC’16), pp.929~934 (2016)
- A. Beygelzimer, "Optimal and adaptive algorithms for online boosting", International Conference on Machine Learning. PMLR (2015)
- Y. Freund, "Boosting a weak learning algorithm by majority", Information and Computation, 121(2), pp.256~285 (1995)
- S. G. T. C. Santos, "Speeding up recovery from concept drifts, in Machine Learning and Knowledge Discovery in Databases", LNCS, 8726, pp.179~194 (2014), Springer
- N. G. Nair, "Covariate shift : A review and analysis on classifiers", 2019 Global Conference for Advancement in Technology (GCAT). IEEE, pp.1~6 (2019)
- H. Shimodaira, "Improving predictive inference under covariate shift by weighting the log-likelihood function", Journal of Statistical Planning and Inference, 90(2), pp.227~244 (2000)
- "Expectation and Variance"[URL]