수질 데이터에 대한 결측값 대치 기법 연구
Missing Value Imputation Technique for Water Quality Dataset
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
많은 연구자들이 다양한 모델을 이용하여 물의 수질을 평가하기 위해 노력하고 있다. 평가 모델에는 결측값이 없는 데이터셋이 필요하지만, 관측 데이터셋에는 결측값이 다수 포함되는 것이현실이다. 단순히 결측값을 삭제하는 방법은 경우에 따라 기저 데이터의 분포를 왜곡시키고 모델의 예측성능에도 편의(bias)를 불러올 위험성이 있다. 본 연구에서는 수질 데이터의 결측값 처리에적합한 기법을 탐색하기 위해, 기존의 KNN과 MICE Imputation, 그리고 생성형 신경망 모델인Autoencoder와 Denoising Autoencoder를 기반으로 몇 가지 대치 기법을 실험하였다. 실험 결과, KNN과 MICE Imputation의 결과를 평균한 Combined Imputation이 실측치에 가장 가깝게 값을 추정하였으며, 이 기법을 적용하여 결측값을 처리한 관측 데이터셋을 support vector machine과ensemble 기반의 분류 모델로 평가한 결과, 결측값을 삭제했을 때에 비해 Accuracy, F1 score, ROC-AUC score, 그리고 MCC(Mathews Correlation Coefficient) 지표가 향상되었다.
Abstract
Many researchers make efforts to evaluate water quality using various models. Such models require a dataset without missing values, but in real world, most datasets include missing values for various reasons. Simple deletion of samples having missing value(s) could distort distribution of the underlying data and pose a significant risk of biasing the model’s inference when the missing mechanism is not MCAR. In this study, to explore the most appropriate technique for handing missing values in water quality data, several imputation techniques were experimented based on existing KNN and MICE imputation with/without the generative neural network model, Autoencoder(AE) and Denoising Autoencoder(DAE). The results shows that KNN and MICE combined imputation without generative networks provides the closest estimated values to the true values. When evaluating binary classification models based on support vector machine and ensemble algorithms after applying the combined imputation technique to the observed water quality dataset with missing values, it shows better performance in terms of Accuracy, F1 score, RoC-AuC score and MCC compared to those evaluated after deleting samples having missing values.
참고문헌 (14)
- "Water and Sanitation"[URL]
- Hyun Kang, "The prevention and handling of the missing data", Korean Journal of Anesthesiol, 64(5), pp.402~406 (2013)[DOI]
- Tlamelo Emmanuel, "A survey on missing data in machine learning", Journal of Big Data, 8(1), pp.140~ (2021)[DOI]
- Jae-Ho Kim, "Performance Comparison of Classification performance Using Water Drinkability Data", Journal of the Korea Institute of Information and Communication Engineering, 27(8), pp.934~940 (2023)[DOI]
- Jinal Patel, "A Machine Learning-Based Water Potability Prediction Model by Using Synthetic Minority Oversampling Technique and Explainable AI", Computational Intelligence and Neuroscience, 2022, pp.9283293~ (2022)[DOI]
- Ratolojanahary Romy, "Model selection to improve multiple imputation for handling high rate missingness in a water quality dataset", Expert Systems With Applications, 131, pp.299~307 (2019)[DOI]
- L. L. Doove, "Recursive partitioning for missing data imputation in the presence of interaction effects", Computational Statistics & Data Analysis, 72, pp.92~104 (2014)[DOI]
- Daniel J. Stekhoven, "MissForest—non-parametric missing value imputation for mixed-type data", Bioinformatics, 28(1), pp.112~118 (2012)[DOI]
- Gondara, L., "MIDA: Multiple Imputation Using Denoising Autoencoders", Advances in Knowledge Discovery and Data Mining. PAKDD 2018 (2018)
- Konstantinos Psychogyios, "Comparison of Missing Data Imputation Methods using the Framingham Heart study dataset", 2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI) (2022)
- Jinsung Yoon, "GAIN : Missing Data Imputation using Generative Adversarial Nets", Proceedings of the 35th International Conference on Machine Learning, 80, pp.5689~5698 (2018)
- Pascal Vincent, "Extracting and Composing Robust Features with Denoising Autoencoders", Proceedings of the 25th International Conference on Machine Learning, pp.1096~1103 (2008)
- "Water Potability Prediction"[URL]