Hadoop Based Wavelet Histogram for Big Data in Cloud
1
조회
0
다운로드
2
인용
0
참고문헌
키워드
초록
Recently, the importance of big data has been emphasized with the development of smartphone, web/SNS. Asa result, MapReduce, which can efficiently process big data, is receiving worldwide attention because of itsexcellent scalability and stability. Since big data has a large amount, fast creation speed, and variousproperties, it is more efficient to process big data summary information than big data itself. Wavelethistogram, which is a typical data summary information generation technique, can generate optimal datasummary information that does not cause loss of information of original data. Therefore, a system applying awavelet histogram generation technique based on MapReduce has been actively studied. However, existingresearch has a disadvantage in that the generation speed is slow because the wavelet histogram is generatedthrough one or more MapReduce Jobs. And there is a high possibility that the error of the data restored by thewavelet histogram becomes large. However, since the wavelet histogram generation system based on theMapReduce developed in this paper generates the wavelet histogram through one MapReduce Job, thegeneration speed can be greatly increased. In addition, since the wavelet histogram is generated by adjustingthe error boundary specified by the user, the error of the restored data can be adjusted from the wavelethistogram. Finally, we verified the efficiency of the wavelet histogram generation system developed in thispaper through performance evaluation.
Abstract
Recently, the importance of big data has been emphasized with the development of smartphone, web/SNS. Asa result, MapReduce, which can efficiently process big data, is receiving worldwide attention because of itsexcellent scalability and stability. Since big data has a large amount, fast creation speed, and variousproperties, it is more efficient to process big data summary information than big data itself. Wavelethistogram, which is a typical data summary information generation technique, can generate optimal datasummary information that does not cause loss of information of original data. Therefore, a system applying awavelet histogram generation technique based on MapReduce has been actively studied. However, existingresearch has a disadvantage in that the generation speed is slow because the wavelet histogram is generatedthrough one or more MapReduce Jobs. And there is a high possibility that the error of the data restored by thewavelet histogram becomes large. However, since the wavelet histogram generation system based on theMapReduce developed in this paper generates the wavelet histogram through one MapReduce Job, thegeneration speed can be greatly increased. In addition, since the wavelet histogram is generated by adjustingthe error boundary specified by the user, the error of the restored data can be adjusted from the wavelethistogram. Finally, we verified the efficiency of the wavelet histogram generation system developed in thispaper through performance evaluation.
참고문헌 (9)
- Y. Matias, "Wavelet-based histograms for selectivity estimation", Proceedings of the ACM SIGMOD International Conference on Management of Data, pp.448~459 (1998)
- E. J. Stollnitz, "Wavelet for Computer Graphics: Theory and Applications" (1996), Morgan Kaufmann
- "MapReduce Tutorial"[URL]
- M. Garofalakis, "Probabilistic wavelet synopses", Journal of ACM Transactions on Database Systems, 29(1), pp.43~90 (2004)
- Y. Shi, "HEDC: a histogram estimator for data in the cloud", Proceedings of the 4th International Workshop on Cloud Data Management, pp.51~58 (2012)
- V. Poosala, "Improved histograms for selectivity estimation of range predicates", Proceedings of the 1996 ACM SIGMOD International Conference on Management of Data, pp.294~305 (1996)
- J. Jestes, "Building wavelet histograms on large data in MapReduce", Proceedings of the VLDB Endowment, 5(2), pp.109~120 (2011)
- P. Cao, "Efficient top-k query calculations in distributed networks", Proceedings of the 23rd Annual ACM Symposium on Principles of Distributed Computing, pp.206~215 (2004)
- "Wikipedia Page Traffic Statistics"[URL]