A Study on Big Data Processing-based Data Concentrated Computation
2
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Over the last decades one could observe a drastic increase in the generation and storage of data in both, industry and science. While the field of data analysis is not new, it is now facing the challenge of coping with an increasing size, bandwidth and complexity of data. This renders traditional analysis methods and algorithms ineffective. This problem has been coined as the Big Data challenge. Concretely in science the major data producers are large-scale monolithic experiments and the outputs of domain simulations. Up until now, most of this data has not yet been completely analyzed, but rather stored in data repositories for later consideration due to the lack of efficient means of processing. We proposes a design and prototypical realization of such a framework based on the experience collected from empirical applications, so we called BDP(Big Data Processing). For this, selected scientific use cases, with an emphasis on earth sciences, were studied. In particular, these are object segmentation in point cloud data and biological imagery, outlier detection in oceanographic time-series data as well as land cover type classification in remote sensing images. In order to deal with the data amounts, two analysis algorithms have been parallelized for shared- and distributed-memory systems. The presented parallelization strategies have been abstracted into a generalized paradigm, enabling the formulation of scalable algorithms for other similar analysis methods. Moreover, it permits a large-scale data analysis framework and algorithm library for heterogeneous, distributed high-performance computing systems.
Abstract
Over the last decades one could observe a drastic increase in the generation and storage of data in both, industry and science. While the field of data analysis is not new, it is now facing the challenge of coping with an increasing size, bandwidth and complexity of data. This renders traditional analysis methods and algorithms ineffective. This problem has been coined as the Big Data challenge. Concretely in science the major data producers are large-scale monolithic experiments and the outputs of domain simulations. Up until now, most of this data has not yet been completely analyzed, but rather stored in data repositories for later consideration due to the lack of efficient means of processing. We proposes a design and prototypical realization of such a framework based on the experience collected from empirical applications, so we called BDP(Big Data Processing). For this, selected scientific use cases, with an emphasis on earth sciences, were studied. In particular, these are object segmentation in point cloud data and biological imagery, outlier detection in oceanographic time-series data as well as land cover type classification in remote sensing images. In order to deal with the data amounts, two analysis algorithms have been parallelized for shared- and distributed-memory systems. The presented parallelization strategies have been abstracted into a generalized paradigm, enabling the formulation of scalable algorithms for other similar analysis methods. Moreover, it permits a large-scale data analysis framework and algorithm library for heterogeneous, distributed high-performance computing systems.
참고문헌 (15)
- 김응성, "Research on Intelligent Dynamic Software Architecture Using Style-based Modeling", 한국지식정보기술학회 논문지, 16(4), pp.685~692 (2021), 한국지식정보기술학회[DOI]
- 김종윤, "Optimized Data Processing Analysis Using Big Data Cloud Platform", 한국지식정보기술학회 논문지, 16(1), pp.1~7 (2021), 한국지식정보기술학회[DOI]
- 조이상, "A Study on Big Data-based GraphX Model for Social Network Service", 한국지식정보기술학회 논문지, 15(6), pp.955~962 (2020), 한국지식정보기술학회[DOI]
- 황성태, "A Study on Big Data Platform Architecture-based Conceptual Measurement Model Using Comparative Analysis for Social Commerce", 한국지식정보기술학회 논문지, 15(5), pp.623~630 (2020), 한국지식정보기술학회[DOI]
- M. Zaharia, "Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing", Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation, pp.2~ (2012), USENIX Association
- C. Wang, "Statistical techniques for online anomaly detection in data centers", Integrated Network Management (IM), 2011 IFIP, 2011, 385~392
- P. Alvaro, "Consistency analysis in bloom: a calm and collected approach", Proceedings 5th Biennial Conference on Innovative Data Systems Research, pp.249~260 (2011)
- N. Pandeeswari, "Anomaly detection system in cloud environment using fuzzy clustering based ANN", Mob. Netw. Appl, 21(3), pp.494~505 (2016)
- M. Jahrer, "Ensemble of collaborative filtering and feature engineered models for click through rate prediction", KDDCup Workshop (2012)
- T. Rabl, "Advancing big data benchmarks, Vol. 8585" (2014), Springer International Publishing
- Sung-Hyun Kim, "Developing a Big Data Analytic Model and a Platform for Particulate Matter Prediction: A Case Study", International Journal of Fuzzy Logic and Intelligent systems, 19(4), pp.242~249 (2019), 한국지능시스템학회
- 박정규, "네트워크 컴퓨팅 기술을 활용한 확장 가능형 빅데이터 스토리지 시스템 개발", 한국정보통신학회논문지, 23(11), pp.1330~1336 (2019), 한국정보통신학회[DOI]
- M. S. Kim, "Designing big data platforms for developing agricultural management strategies", Magazine of the Korean Society of Agricultural Engineers, 59(4), pp.26~34 (2017)
- M. Al-Mekhal, "A synthesis of big data definition and characteristics", 2019 IEEE International Conference on Computational Science and Engineering (CSE) and IEEE International Conference on Embedded and Ubiquitous Computing (EUC), pp.314~322 (2019)
- X. W. Sha, "AI & Society: Knowledge Culture and Communication", pp.1~4 (2016)