Optimizing NUMA System Beyond 100 Gbps for Big Data
2
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
As big data science and analysis become prevalent in the science community and industry, systems to store, access, and share big data are studied in many ways. As compute resources to execute big data analysis move to the cloud, the storage system integrates to the cloud to provide storage services to the compute resources. The accessibility of big data is important to cloud computing because the analysis jobs that run in the cloud often migrate to another compute node. The storage service needs to provide the resource to multiple compute nodes with NUMA support. Current High-performance data management systems focus on the accessibility of storage systems and lack of performance optimization for transferring data in NUMA system. Big data and cloud computing's popularity enhance the trend of universally accessible data, yet accessing remote data is limited significantly due to the lack of optimization on the data path within the systems and the networks. Our systematic approach to optimize a NUMA system provides fully utilizing end-to-end data transfer streams with 100 Gbps disk-to-disk throughput. We use our model to show we can utilize the 100 Gbps network with disk-to-disk transfer using commodity hardware and open-source tools with optimizing NUMA system.
Abstract
As big data science and analysis become prevalent in the science community and industry, systems to store, access, and share big data are studied in many ways. As compute resources to execute big data analysis move to the cloud, the storage system integrates to the cloud to provide storage services to the compute resources. The accessibility of big data is important to cloud computing because the analysis jobs that run in the cloud often migrate to another compute node. The storage service needs to provide the resource to multiple compute nodes with NUMA support. Current High-performance data management systems focus on the accessibility of storage systems and lack of performance optimization for transferring data in NUMA system. Big data and cloud computing's popularity enhance the trend of universally accessible data, yet accessing remote data is limited significantly due to the lack of optimization on the data path within the systems and the networks. Our systematic approach to optimize a NUMA system provides fully utilizing end-to-end data transfer streams with 100 Gbps disk-to-disk throughput. We use our model to show we can utilize the 100 Gbps network with disk-to-disk transfer using commodity hardware and open-source tools with optimizing NUMA system.
참고문헌 (17)
- S. Y. Yu, "Analysis of CPU pinning and storage configuration in 100 Gbps network data transfer", IEEE, 2018, ~
- J. Postel, "Transmission control protocol, ietf RFC 793", ~ (1981)
- N. Cardwell, "BBR:congestion-based congestion control", ACM Queue, 14, pp.20~53 (2016)
- G. Vardoyan, "Characterizing throughput bottlenecks for secure GridFTP transfers", International Conference on Computing, Networking and Communications (2013)
- Q. Lu, "Bigdata express: toward schedulable, predictable, and high-performance data transfer", IEEE, 2018, ~
- Y. Gu, "UDT : UDP-based data transfer for high-speed wide area networks", Computer Networks, 51(7), pp.1777~1799 (2007)
- I. Foster, "Globus online : accelerating and democratizing science through cloud-based services", IEEE Internet Computing, 15(3), pp.70~73 (2011)
- R. Kettimuthu, "Globus xio pipe open driver: enabling gridftp to leverage standard unix tools", TeraGrid Conference: Extreme Digital Discovery (2011)
- A. Dorigo, "Xrootd", txnetfile: a highly scalable architecture for data access in the root environment, 2005, ~
- 최원근, "SCTP에서 ECMC 기법의 전송 지연에 관한 연구", 한국지식정보기술학회 논문지, 12(4), pp.513~519 (2017), 한국지식정보기술학회[DOI]
- 김종윤, "Optimized Data Processing Analysis Using Big Data Cloud Platform", 한국지식정보기술학회 논문지, 16(1), pp.1~7 (2021), 한국지식정보기술학회[DOI]
- N. Manchanda, "Non-uniform memory access(numa)" (2010), New York University
- G. M. Kurtzer, "Singularity: Scientific containers for mobility of compute", PLOS ONE, 12(5), pp.1~20 (2017)
- D. Bernstein, "Containers and cloud : from lxc to docker to kubernetes", IEEE Cloud Computing, 1(3), pp.81~84 (2014)
- R. Peinl, "Docker cluster management for the cloud -survey results and own solution", Journal of Grid Computing, 14(2), pp.265~282 (2016)
- N. Naik, "Building a virtual system of systems using docker swarm in multiple clouds", IEEE International Symposium on Systems Engineering, pp.1~3 (2016)
- K. T. Murata, "A bandwidth challenge at super computing (sc) conference:large-scale data transfer using 10gbps network", Optical Fiber Communication Conference and National Fiber Optic Engineers Conference (2009)