도커 기반의 실시간 데이터 연계 및 처리 환경을 고려한 빅데이터 관리 플랫폼 개발
Development of Big-data Management Platform Considering Docker Based Real Time Data Connecting and Processing Environments
2
조회
0
다운로드
3
인용
0
참고문헌
키워드
초록
Real-time access is required to handle continuous and unstructured data and should be flexible in management under dynamic state. Platform can be built to allow data collection, storage, and processing from local-server or multi-server. Although the former centralize method is easy to control, it creates an overload problem because it proceeds all the processing in one unit, and the latter distributed method performs parallel processing, so it is fast to respond and can easily scale system capacity, but the design is complex. This paper provides data collection and processing on one platform to derive significant insights from various data held by an enterprise or agency in the latter manner, which is intuitively available on dashboards and utilizes Spark to improve distributed processing performance. All service utilize dockers to distribute and management. The data used in this study was 100% collected from Kafka, showing that when the file size is 4.4 gigabytes, the data processing speed in spark cluster mode is 2 minute 15 seconds, about 3 minutes 19 seconds faster than the local mode.
Abstract
Real-time access is required to handle continuous and unstructured data and should be flexible in management under dynamic state. Platform can be built to allow data collection, storage, and processing from local-server or multi-server. Although the former centralize method is easy to control, it creates an overload problem because it proceeds all the processing in one unit, and the latter distributed method performs parallel processing, so it is fast to respond and can easily scale system capacity, but the design is complex. This paper provides data collection and processing on one platform to derive significant insights from various data held by an enterprise or agency in the latter manner, which is intuitively available on dashboards and utilizes Spark to improve distributed processing performance. All service utilize dockers to distribute and management. The data used in this study was 100% collected from Kafka, showing that when the file size is 4.4 gigabytes, the data processing speed in spark cluster mode is 2 minute 15 seconds, about 3 minutes 19 seconds faster than the local mode.
참고문헌 (26)
- H. R. Yu, "2020 Data Industrial White-paper : Market Status", Korea Data Agency, (23), pp.110~141 (2020)
- 손형섭, "디지털 뉴딜 관련 데이터 법제의 입법 방향에 관한 연구", 헌법학연구, 27(1), pp.203~252 (2021), 한국헌법학회
- 김학래, "국가 데이터의 의미적 표현과 연계를 위한 데이터맵 지식 모델", 디지털콘텐츠학회논문지, 22(3), pp.491~499 (2021), 한국디지털콘텐츠학회[DOI]
- 방정미, "공공데이터 활용 실재화를 위한 데이터기반행정", 비교법연구, 21(1), pp.87~126 (2021), 비교법문화연구소[DOI]
- 천승만, "제조로봇 고장예지진단을 위한 오픈소스기반 스마트 제조 빅데이터 플랫폼 구현", 대한임베디드공학회논문지, 14(4), pp.187~195 (2019), 대한임베디드공학회[DOI]
- D. G. Kim, "A Study on Bigdata Collection and Processing ENgine of Docker Based", Korea Institute of Intelligent Transportation Systems Proceedings of the Spring Conference, pp.593~594 (2021)
- P. Hoenisch, "Four-Fold Auto-Scaling on a Contemporary Deployment Platform Using Docker Containers", Lecture Notes in Computer Science, 2015(9435), pp.316~323 (2015)
- A. Hosny, "AlgoRun : a Docker-based Packaging System for Platform-agnostic Implemented Algorithms", Bioinformatics, 32(15), pp.2396~2398 (2016)
- A. Eiermann, "On a Fog Computing Platform Built on ARM Architectures by Docker Container Technology", Communications in Computer and Information Science, 2017(717), pp.71~86 (2017)
- C. W. Tien, "KubAnomaly : Anomaly Detection for the Docker Orchestration Platform with Neural Network Approaches", Engineering reports, 1(5), pp.1~20 (2019)
- 장래영, "OpenFaaS 기반 AI 분석 서비스 시스템 구축", 한국콘텐츠학회 논문지, 20(7), pp.97~106 (2020), 한국콘텐츠학회[DOI]
- 김태영, "실시간 자원 모니터링을 활용한 쿠버네티스 스케줄러 프레임워크 구현", 대한임베디드공학회논문지, 15(3), pp.129~137 (2020), 대한임베디드공학회[DOI]
- I. S. Jeong, "Apache Kafka : From Application Development to Pipeline, Internet of Things data hub Construction", pp.1~388 (2020), Hanbit Media
- 류우석, "오픈소스 플랫폼 기반의 실시간 환자 대기시간 모니터링 시스템 설계", 한국정보통신학회논문지, 22(4), pp.575~580 (2018), 한국정보통신학회[DOI]
- 명노영, "아파치 스파크 활용 극대화를 위한 성능 최적화 기법", 정보처리학회논문지. 컴퓨터 및 통신시스템, 7(1), pp.9~18 (2018), 한국정보처리학회[DOI]
- 고세윤, "Apache Spark를 활용한 대용량 데이터의 처리", 응용통계연구, 29(6), pp.1077~1094 (2016), 한국통계학회
- S. M. Baek, "Spark Two Programming for the Bigdata Analysis", pp.1~630 (2018), Wikibooks
- "software"[URL]
- 한동희, "다중 웹사이트 관리를 위한 동작 기반 웹 크롤러 설정 구조 설계", 정보과학회 컴퓨팅의 실제 논문지, 27(2), pp.98~103 (2021), 한국정보과학회[DOI]