BigCrawler: 엣지 서버 컴퓨팅·스토리지 모듈의 동적 구성을 통한 효율적인 빅데이터 처리 시스템 구현 및 성능 분석
Implementation and Performance Aanalysis of Efficient Big Data Processing System Through Dynamic Configuration of Edge Server Computing and Storage Modules
2
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Edge Computing enables real-time big data processing by performing computing close to the physical location of the user or data source. However, in an edge computing environment, various situations that affect big data processing performance may occur depending on temporary service requirements or changes of physical resources in the field. In this paper, we proposed a BigCrawler system that dynamically configures the computing module and storage module according to the big data collection status and computing resource usage status in the edge computing environment. And the feature of big data processing workload according to the arrangement of computing module and storage module were analyzed.
Abstract
Edge Computing enables real-time big data processing by performing computing close to the physical location of the user or data source. However, in an edge computing environment, various situations that affect big data processing performance may occur depending on temporary service requirements or changes of physical resources in the field. In this paper, we proposed a BigCrawler system that dynamically configures the computing module and storage module according to the big data collection status and computing resource usage status in the edge computing environment. And the feature of big data processing workload according to the arrangement of computing module and storage module were analyzed.
참고문헌 (17)
- J. Wang, "Industrial Big Data Analytics : Challenges, Methodologies, and Applications", CoRR, arXiv : 1807. 01016, ~ (2018)
- B. Duncan, "Enterprise security and privacy: Why adding IoT and big data makes it so much more difficult", International Conference on Engineering and Technology (ICET), pp.1~7 (2017)
- M. Caprolu, "Edge Computing Perspectives: Architectures, Technologies, and Open Security Issues", IEEE International Conference on Edge Computing (EDGE), pp.116~123 (2019)
- "Technology Roadmap of SME"[URL]
- Gigabyte, "Edge Server", ~ (2021)
- Dell, "Dell EMC PowerEdge XE Servers", ~ (2021)
- 김동길, "도커 기반의 실시간 데이터 연계 및 처리 환경을 고려한 빅데이터 관리 플랫폼 개발", 대한임베디드공학회논문지, 16(4), pp.153~161 (2021), 대한임베디드공학회[DOI]
- 김태현, "저지연 서비스를 위한 Multi-access Edge Computing 스케줄러", 대한임베디드공학회논문지, 15(6), pp.299~305 (2020), 대한임베디드공학회[DOI]
- 신승준, "스마트공장을 위한 빅데이터 애널리틱스 플랫폼 아키텍쳐 개발", 멀티미디어학회논문지, 19(8), pp.1516~1529 (2016), 한국멀티미디어학회[DOI]
- "Kubernetes"[URL]
- "Apache Lucene"[URL]
- "Elastic(ELK)Stack"[URL]
- "Web Data Crawling vs Web Data Scraping"[URL]
- "Prometheus"[URL]
- "Analytics end-to-end with Azure Synapse"[URL]
- "AWS serverless data analytics pipeline reference architecture"[URL]
- "Power facility energy pattern and failure analysis sensor"[URL]