Distributed Moving Objects Management System for a Smart Black Box
2
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
In this paper, we design and implement a distributed, moving objects management system for processing locations and sensor data from smart black boxes. The proposed system is designed and implemented based on Apache Kafka, Apache Spark & Spark Streaming, Hbase, HDFS. Apache Kafka is used to collect the data from smart black boxes and queries from users. Received location data from smart black boxes and queries from users becomes input of Apache Spark Streaming. Apache Spark Streaming preprocesses the input data for indexing. Recent location data and indexes are stored in-memory managed by Apache Spark. Old data and indexes are flushed into HBase later. We perform experiments to show the throughput of the index manager. Finally, we describe the implementation detail in Scala function level.
Abstract
In this paper, we design and implement a distributed, moving objects management system for processing locations and sensor data from smart black boxes. The proposed system is designed and implemented based on Apache Kafka, Apache Spark & Spark Streaming, Hbase, HDFS. Apache Kafka is used to collect the data from smart black boxes and queries from users. Received location data from smart black boxes and queries from users becomes input of Apache Spark Streaming. Apache Spark Streaming preprocesses the input data for indexing. Recent location data and indexes are stored in-memory managed by Apache Spark. Old data and indexes are flushed into HBase later. We perform experiments to show the throughput of the index manager. Finally, we describe the implementation detail in Scala function level.
참고문헌 (13)
- H. Li, "Grid based Distributed Inmemory Indexing for Moving Objects", Proceedings of International Symposium on Information Technology Convergence (2014)
- K. Kim, "Optimizing Multidimensional Index Trees for Main Memory Access", SIGMOD Record, 30(2), pp.139~150 (2001)[DOI]
- L. Biveinis, "Main-memory Operation Buffering for Efficient R-tree Update", Proceedings of the 33rd 41st International Conference on Very Large Data Bases, pp.591~602 (2007)
- D. Sidlauskas, "Parallel Mainmemory Indexing for Moving-object Query and Update Workloads", Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, pp.37~48 (2012)
- S. Nishimura, "MD-HBase: A Scalable Multi-dimensional Data Infrastructure for Location Aware Services", Proceedings of the 2011 12th IEEE International Conference on Mobile Data Management, pp.7~16 (2011)
- A. Aji, "Hadoop-GIS: A High Performance Spatial Data Warehousing System over Mapreduce", Proceedings of the 39rd 41st International Conference on Very Large Data Bases, pp.1009~1020 (2013)
- A. Eldawyand, "Spatial Hadoop: A Mapreduce Framework for Spatial Data", Proceedings of the 2015 IEEE 31st International Conference on Data Engineering, pp.1352~1363 (2015)
- J. Lu, "Parallel Secondo: A Practical System for Largescale Processing of Moving Objects", Proceedings of the 2014 IEEE 30st International Conference on Data Engineering, pp.1190~1193 (2014)
- A. R. Mahmood, "Tornado: A Distributed Spatiotextual Stream Processing System", Proceedings of the 41st International Conference on Very Large Data Bases, pp.2020~2023 (2015)
- "Apache Spark"[URL]
- "Apache Hadoop"[URL]
- "Apache Storm"[URL]
- H. Lee, "Implementation of Distributed In-Memory Moving Objects Management System", Advanced Science Letters, 23(10), pp.10361~10365 (2017)[DOI]