Real-Time Processing System of E-Commerce User Data Based on Spark Streaming
2
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
The advent of the e-commerce era has changed the way people shop, and at the same time, users generate a large amount of data when shopping. These data can be analyzed by offline calculation, but the results of offline analysis lack real-time performance. In this paper, by processing the log data and business data of e-commerce users in real-time, the feedback of the processing results can be quickly realized. The Spark big data computing framework has the advantages of real-time computing capability and high throughput. Spark Streaming, as an extension of Spark core, is the real-time stream processing component of the Spark computing platform. In this paper, the data is processed in real-time through Spark. Through Maxwell, real-time monitoring of business data changes in the MySQL database is performed, and the monitored data is sent to Kafka. Log data is directly sent to Kafka. Spark Streaming consumes the data in Kafka, then performs specific processing on the data according to the requirements, and the processed data is written to the Elasticsearch. In order to achieve exactly once consumption of data, this paper realizes at least one consumption of data by manually submitting offsets. Elasticsearch supports idempotent writes, so it can achieve exactly once consumption of downstream data. Manually submitted offsets are stored in Redis. Finally, specific queries can be performed on the processing results according to business requirements.
Abstract
The advent of the e-commerce era has changed the way people shop, and at the same time, users generate a large amount of data when shopping. These data can be analyzed by offline calculation, but the results of offline analysis lack real-time performance. In this paper, by processing the log data and business data of e-commerce users in real-time, the feedback of the processing results can be quickly realized. The Spark big data computing framework has the advantages of real-time computing capability and high throughput. Spark Streaming, as an extension of Spark core, is the real-time stream processing component of the Spark computing platform. In this paper, the data is processed in real-time through Spark. Through Maxwell, real-time monitoring of business data changes in the MySQL database is performed, and the monitored data is sent to Kafka. Log data is directly sent to Kafka. Spark Streaming consumes the data in Kafka, then performs specific processing on the data according to the requirements, and the processed data is written to the Elasticsearch. In order to achieve exactly once consumption of data, this paper realizes at least one consumption of data by manually submitting offsets. Elasticsearch supports idempotent writes, so it can achieve exactly once consumption of downstream data. Manually submitted offsets are stored in Redis. Finally, specific queries can be performed on the processing results according to business requirements.
참고문헌 (15)
- 정원일, "스파크 프레임워크에서 연속집계 질의처리를 위한 자원공유기법", 한국지식정보기술학회 논문지, 15(5), pp.671~681 (2020), 한국지식정보기술학회[DOI]
- 김종윤, "Optimized Data Processing Analysis Using Big Data Cloud Platform", 한국지식정보기술학회 논문지, 16(1), pp.1~7 (2021), 한국지식정보기술학회[DOI]
- Y. S. Jeong, "A token-based authentication security scheme for hadoop distributed file system using elliptic curve cryptography", Journal of computer virology and hacking techniques, pp.137~142 (2015)
- 장도, "Personalized Product Recommendation System Based on Spark", 한국지식정보기술학회 논문지, 17(5), pp.795~805 (2022), 한국지식정보기술학회[DOI]
- M. Assefi, "Big data machine learning using apache spark MLlib", 2017 IEEE international conference on big data, pp.3492~3498 (2017)
- E. D. Hemdan, "Spark-based log data analysis for reconstruction of cybercrime events in cloud environment", 2017 international conference on circuit, power and computing technologies(ICCPCT), pp.1~8 (2017), IEEE
- J. Wei, "A research on smart tourism-oriented big data real-time processing technology", 2017 29th chinese control and decision conference(CCDC), pp.1848~1851 (2017), IEEE
- B. Hiraman, "A study of apache kafka in big data stream processing", 2018 International Conference on Information, Communication, engineering and technology(ICICET), pp.1~3 (2018), IEEE
- C. A. Baron, "NoSQL key-value DBs riak and redis", Database systems journal, pp.3~10 (2016)
- V. A. Zamfir, "Systems monitoring and big data analysis using the elasticsearch system", 2019 22nd international conference on control systems and computer science(CSCS), pp.188~193 (2019), IEEE
- A. K. Rastogi, "Ambient communications and computer systems", pp.293~303 (2018), springer
- A study on modern messaging systems-kafka, rabbitmq and nats streaming, "T. Sharvari", CoRR, ~, pp.CoRR abs (2019), 1912.03715
- D. Patel, "Towards in-order and exactly-once delivery using hierarchical distributed message queues", 2014 14th IEEE, 2014, 883~892[DOI]
- B. Namdeo, "A model for relational to NoSQL database migration:snapshot-live stream DB migration model", 2021 7th international conference on advanced computing and communication systems(ICACCS), pp.199~204 (2021), IEEE
- 편도길, "객체 탐지를 이용한 블루베리 생육량 측정 시스템", 한국지식정보기술학회 논문지, 17(3), pp.465~474 (2022), 한국지식정보기술학회[DOI]