빅데이터를 위한 데이터 품질 평가 방법에 대한 연구
A Survey of Data Quality Assessment Methods for Big Data
0
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
빅데이터 분석이나 머신러닝과 같은 데이터 기반의 정보기술 분야에서는 데이터 품질이 매우 중요하다. 고성능의 데이터 분석 알고리즘이나 머신러닝 학습 모델을 사용하더라도 입력 데이터의 품질이 보장되지 않는다면, 그 결과물은 신뢰할 수 없기 때문이다. 따라서 빅데이터를 활용하고 분석하기 위해서는 방대하고 복잡한 데이터들로부터 고품질의 데이터를 추출할 수 있어야 한다. 본 논문에서는 고품질을 보장하는 빅데이터를위한 품질 평가 방법에 대해 고찰해본다. 빅데이터에 대한 데이터 라이프사이클을 살펴보고, 데이터 품질 요소에 대한 국제표준화 동향과 함께 라이프 사이클에 따라 고려되어야 하는 데이터 품질 특성을 정의한다. 또한 빅데이터 관련 데이터 품질 평가에 대한 기존의 주요 연구들을 비교 분석하고, 이를 바탕으로 빅데이터품질 평가에 필요한 요소들을 살펴본다. 정의된 요소들을 반영하여 목적기반 데이터 품질지표를 이용한 빅데이터 품질 평가 프로세스를 제안한다. 제안된 프로세스는 향후 빅데이터 품질을 평가할 수 있는 평가 지표개발과 통합된 평가 프레임워크 개발의 기초자료로 활용될 것으로 기대한다.
Abstract
Data quality is very important in data-based information technologies such as big data analysis or machine learning. Even if high-performance data analysis algorithms or machine learning models are used, if the quality of the input data is not guaranteed, the results cannot be trusted. Therefore, in order to utilize and analyze big data, it is necessary to be able to extract high-quality data from massive and complex data. In this paper, we consider quality evaluation methods for big data that guarantee high quality. We examine international standardization trends for data life cycle and data quality factors for big data and define data quality characteristics that should be considered according to the big data life cycle. In addition, we compare and analyze existing major studies on data quality evaluation related to big data, and based on the result, we examine the elements necessary for big data quality evaluation. Based on the defined elements, we propose a big data quality evaluation process using goal-driven data quality metric. In the future, we expect that the proposed process will be used to develop evaluation indicators that can evaluate big data quality and to develop an integrated evaluation framework.
참고문헌 (17)
- Suwook Ha, "Data quality standardization for data analysis and machine learning", TTA Journal, 192, pp.22~27 (2020)
- Curtis Northcutt, "Confident Learning: Estimating Uncertainty in Dataset Labels", Journal of Artificial Intelligence Research, 70, pp.1373~1411 (2021), AI Access Foundation[DOI]
- O. Reda, "Towards a Data Quality Assessment in Big Data", Proceedings of the International Conference on Intelligent Systems: Theories and Applications, pp.1~6 (2020)
- Diane M. Strong, "Data quality in context", Communications of the ACM, 40(5), pp.103~110 (1997), Association for Computing Machinery (ACM)[DOI]
- "ITU-T Y.3604, Big data – Requirements and conceptual model of metadata for data catalogue" (2019)[URL]
- Youdi Gong, "A survey on dataset quality in machine learning", Information and Software Technology, 162, pp.107268~ (2023), Elsevier BV[DOI]
- "ISO", IEC 25012, Software engineering –Software product quality requirements and evaluation (SQuaRE) - Data quality model, 2008
- "ISO/IEC DIS 5259" (2023)[URL]
- "ISO/IEC JTC1/SC 42(AI)/WG2(Data) Data Qulaity for Analytics and Machine Learning (ML)" (2022)[URL]
- I. Taleb, "Big data quality: A survey", Proceeding of IEEE International congress on Big Data, pp.166~173 (2018)
- Li Ya, "RETRACTED: Studies on Data Quality Evaluation Index System for Internet Plus Government Services in Big Data Era", Journal of Physics: Conference Series, 1584(1), pp.012014~ (2020), IOP Publishing[DOI]
- Hong Chen, "A Review of Data Quality Assessment Methods for Public Health Information Systems", International Journal of Environmental Research and Public Health, 11(5), pp.5170~5207 (2014), MDPI AG[DOI]
- S. Chug, "Statistical learning to operationalize a domain agnostic data quality scoring" (2021)[DOI]
- Li Cai, "The Challenges of Data Quality and Data Quality Assessment in the Big Data Era", Data Science Journal, 14(2) (2015), Ubiquity Press, Ltd.[DOI]
- Christian Bors, "Visual Interactive Creation, Customization, and Analysis of Data Quality Metrics", Journal of Data and Information Quality, 10(1), pp.1~26 (2018), Association for Computing Machinery (ACM)[DOI]
- Fakhitah Ridzuan, "A Review on Data Cleansing Methods for Big Data", Procedia Computer Science, 161, pp.731~738 (2019), Elsevier BV[DOI]
- Mohamad Al Jazaery, "Automated cleaning of identity label noise in a large face dataset with quality control", IET Biometrics, 9(1), pp.25~30 (2019), Institution of Engineering and Technology (IET)[DOI]