맵리듀스에서 집계 질의 스트림의 효율적인 처리 기법
Efficient Processing of an Aggregate Query Stream in MapReduce
0
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
맵리듀스는 빅데이터 분석 및 처리에 널리 사용되는 프로그래밍 모델이다. 빅데이터 분석을 위해 흔히 사용되는 질의 중 하나는 집계 질의 (aggregate query)이다. 본 논문에서는 여러 사용자가 동시에 여러 집계 질의를 계속해서 요청하는 경우, 맵리듀스를 사용하여 이들 질의를 효율적으로 처리하는 방법을 제안한다. 제안 방법은 각 집계 질의를 개별적으로 처리하지 않고, 여러 집계 질의를 묶어 하나의 최적화된 맵리듀 스 잡(job)으로 만들어 일괄 처리한다. 그 결과로 제안 방법은 단순 방법에 비해 시간당 처리하는 질의 수를 크게 증가시킨다. 성능 평가를 통해, 제안 방법은 단순 방법에 비해 질의 처리 속도를 크게 향상시킴을 보인다.
Abstract
MapReduce is a widely used programming model for analyzing and processing Big data. Aggregate queries are one of the most common types of queries used for analyzing Big data. In this paper, we propose an efficient method for processing an aggregate query stream, where many concurrent users continuously issue different aggregate queries on the same data. Instead of processing each aggregate query separately, the proposed method processes multiple aggregate queries together in a batch by a single, optimized MapReduce job. As a result, the number of queries processed per unit time increases significantly. Through various experiments, we show that the proposed method improves the performance significantly compared to a naive method.
참고문헌 (21)
- Mark Beyer, "Gartner Says Solving ‘Big Data’ Challenge Involves More Than Just Managing Volumes of Data", Gartner, ~ (2011)
- Jeffrey Dean, "MapReduce: simplified data processing on large clusters", In Proceedings of OSDI '04, pp.137~150 (2004)
- Hyunjean Choi, "Efficient Processing of an Aggregate Query Stream in MapReduce", Korea Information Processing Society Fall Conference (2013)
- Spyros Blanas, "A comparison of join algorithms for log processing in MaPreduce", In Proceedings of the 2010 ACM SIGMOD International Conference on Management of data, pp.975~986 (2010)
- Alper Okcan, "ACM SIGMOD International Conference on Management of data, pp", In Proceedings of the 2011 ACM SIGMOD International Conference on Management of data, pp.949~960 (2011)
- Foto N. Afrati, "Optimizing Multiway Joins in a Map-Reduce Environment", IEEE Transactions on Knowledge and Data Engineering, 23(9), pp.1282~1298 (2011)
- "Hive"[URL]
- "HadoopDB"[URL]
- Tyson Condie, "MapReduce Online", In Proceedings of NSDI, 10(4), pp.20~ (2010)
- Pramod Bhatotia, "Incoop: MapReduce for Incremental Computations", In Proceedings of SOCC'11 (2011)
- Dionysios Logothetis, "In-situ MapReduce for log processing", In Proceedings of USENIXATC'11 (2011)
- Wang Lam, "Muppet: MapReduce-style processing of fast data", In Proceedings of the VLDB Endowment, 5(12), pp.1814~1825 (2012)
- Tomasz Nykiel, "MRShare: sharing across multiple queries in MapReduce", In Proceedings of the VLDB Endowment, 3(1-2), pp.494~505 (2010)
- Guoping Wang, "Multi-Query Optimization in MapReduce Framework", In Proceedings of the VLDB Endowment, 7(3), pp.145~156 (2013)
- "Cloudera Impala"[URL]
- "Tajo"[URL]
- Dmitri V. Kalashnikov, "Main Memory Evaluation of Monitoring Queries Over Moving Objects", Distributed and Parallel Databases, 15(2), pp.117~135 (2004)
- Xiaohui Yu, "Monitoring k-Nearest Neighbor Queries Over Moving Objects", In Proceedings of the 21st International Conference on Data Engineering, pp.631~642 (2005)
- "Amazon Elastic Compute Cloud(Amazon EC2)"[URL]