Optimized Data Processing Analysis Using Big Data Cloud Platform
2
조회
0
다운로드
8
인용
0
참고문헌
키워드
초록
Recently data processing has main gained many attention both from academic and commercial industry. Their term use to tools, technical methods and frameworks made to gathering, collect, store, processing and analysis massive amounts of data. Data processing and analysis are based on structured/semi -structured/unstructured by big data, as well as is generated from various different sources in the system at various rates. For the purpose of processing with their large data and suitable way, voluminous parallelism is usually used. The general architecture of a big data system is made up a shared cluster of their machines. Nonetheless, even in very parallel environment, data processing is often very time-consuming. A various applications can take up to everytime to produce useful results, interactive analysis and debugging. Nevertheless, we have main problems that how we have a high performance requires both quality of data locality and resource. Moreover, big data analysis provide the amount of data that is processed typically large in comparison with computation in their systems. In other words, specified optimization that would relieve low-level to achieve good performance essentially. Accordingly, our main goal of this research paper provide how we have to do to optimize for big data frameworks. Our contribute approach to make big data cloud platform for easy and efficient processing of big data. In addition, we provides results from a study of existing optimization of data processing in MapReduce and Hadoop oriented systems.
Abstract
Recently data processing has main gained many attention both from academic and commercial industry. Their term use to tools, technical methods and frameworks made to gathering, collect, store, processing and analysis massive amounts of data. Data processing and analysis are based on structured/semi -structured/unstructured by big data, as well as is generated from various different sources in the system at various rates. For the purpose of processing with their large data and suitable way, voluminous parallelism is usually used. The general architecture of a big data system is made up a shared cluster of their machines. Nonetheless, even in very parallel environment, data processing is often very time-consuming. A various applications can take up to everytime to produce useful results, interactive analysis and debugging. Nevertheless, we have main problems that how we have a high performance requires both quality of data locality and resource. Moreover, big data analysis provide the amount of data that is processed typically large in comparison with computation in their systems. In other words, specified optimization that would relieve low-level to achieve good performance essentially. Accordingly, our main goal of this research paper provide how we have to do to optimize for big data frameworks. Our contribute approach to make big data cloud platform for easy and efficient processing of big data. In addition, we provides results from a study of existing optimization of data processing in MapReduce and Hadoop oriented systems.
참고문헌 (16)
- 조이상, "A Study on Big Data-based GraphX Model for Social Network Service", 한국지식정보기술학회 논문지, 15(6), pp.955~962 (2020), 한국지식정보기술학회[DOI]
- 황성태, "A Study on Big Data Platform Architecture-based Conceptual Measurement Model Using Comparative Analysis for Social Commerce", 한국지식정보기술학회 논문지, 15(5), pp.623~630 (2020), 한국지식정보기술학회[DOI]
- 김정식, "A Study on Adaptive Smart Platform for Intelligent Software in Big Data Environment", 한국지식정보기술학회 논문지, 15(3), pp.347~355 (2020), 한국지식정보기술학회[DOI]
- H. S. Munawar, "Big data and its applications in smart real estate and the disaster management life cycle: a systematic analysis, big data cognitive", Computing, MDPI Journal, 4(2) (2020)
- A. Siddiqa, "A survey of big data management: taxonomy and state-of-the-art", Journal of Network and Computer Applications, 71, pp.151~166 (2016), Elsevier
- I. Yaqoob, "Big data: From beginning to future", International Journal of Information Management, 36(6), pp.1231~1247 (2016), Elsevier
- Sung-Hyun Kim, "Developing a Big Data Analytic Model and a Platform for Particulate Matter Prediction: A Case Study", International Journal of Fuzzy Logic and Intelligent systems, 19(4), pp.242~249 (2019), 한국지능시스템학회
- R. Singh, "Analyzing performance of apache tez and MapReduce with hadoop multinode cluster on amazon cloud", Journal of Big Data, SpingerOpen Journal, 3(19) (2016)
- M. Zaharia, "Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing", Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation, pp.2~ (2012), USENIX Association
- C. Wang, "Statistical techniques for online anomaly detection in data centers", Integrated Network Management (IM), 2011 IFIP, 2011, 385~392
- B. Agrawal, "R2Time: A framework to analyse open time-series data in HBase", Cloud Computing Technology and Science (CloudCom), 2014 IEEE 6th International Conference, pp.970~975 (2014)
- T. Akidau, "The dataflow model: a practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing", Proceeding VLDB Endow, 8(12), pp.1792~1803 (2015)
- P. Alvaro, "Consistency analysis in bloom: a calm and collected approach", Proceedings 5th Biennial Conference on Innovative Data Systems Research, pp.249~260 (2011)
- N. Pandeeswari, "Anomaly detection system in cloud environment using fuzzy clustering based ANN", Mob. Netw. Appl, 21(3), pp.494~505 (2016)
- M. Jahrer, "Ensemble of collaborative filtering and feature engineered models for click through rate prediction", KDDCup Workshop (2012)
- T. Rabl, "Advancing Big Data Benchmarks, Vol. 8585" (2014), Springer International Publishing