Performance of NVMe and NVMe-over-Fabrics in NUMA System with High-Performance Network
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
To find a common bottleneck in Data Transfer Nodes using modern NUMA systems, we need to investigate many components in the system. Even efficient modern data transfer systems suffer from performance degradation when used in high-speed data transfer because of the large amount of process load on the system, especially on the CPU and the storage devices. Many efficient storage systems provide a particular benefit in a specific situation. However, there is no comprehensive study of these protocols. There are proposed techniques to optimize CPU load on data transfer and configure storage systems, such as using different CPU core affinity bindings in their Non-Uniform Memory Access (NUMA) system NVMe-over-Fabrics to avoid CPU bottleneck in high-speed data transfer. Such techniques limit the user processes to specific NUMA nodes to reduce foreign memory access overhead. However, this results from a smaller number of available CPU cores in the NUMA system, which is counter-intuitive to run a large workload accessing multiple storage devices. To evaluate the performance of local file systems in high-speed data transfer and CPU affinity binding in a high-speed data transfer system, we performed an analysis of storage and CPU affinity binding in a 100 Gbps network. We achieved the maximum SSD performance threshold using 32 transfer processes with traditional file transfer while using one process per NVMe with NVMe-over-Fabrics and reduced CPU utilization. We could not find significant evidence that binding processes to the local processor or cores improve the file transfer performance with NVMe-over-Fabrics.
Abstract
To find a common bottleneck in Data Transfer Nodes using modern NUMA systems, we need to investigate many components in the system. Even efficient modern data transfer systems suffer from performance degradation when used in high-speed data transfer because of the large amount of process load on the system, especially on the CPU and the storage devices. Many efficient storage systems provide a particular benefit in a specific situation. However, there is no comprehensive study of these protocols. There are proposed techniques to optimize CPU load on data transfer and configure storage systems, such as using different CPU core affinity bindings in their Non-Uniform Memory Access (NUMA) system NVMe-over-Fabrics to avoid CPU bottleneck in high-speed data transfer. Such techniques limit the user processes to specific NUMA nodes to reduce foreign memory access overhead. However, this results from a smaller number of available CPU cores in the NUMA system, which is counter-intuitive to run a large workload accessing multiple storage devices. To evaluate the performance of local file systems in high-speed data transfer and CPU affinity binding in a high-speed data transfer system, we performed an analysis of storage and CPU affinity binding in a 100 Gbps network. We achieved the maximum SSD performance threshold using 32 transfer processes with traditional file transfer while using one process per NVMe with NVMe-over-Fabrics and reduced CPU utilization. We could not find significant evidence that binding processes to the local processor or cores improve the file transfer performance with NVMe-over-Fabrics.
참고문헌 (15)
- A. Huffman, "The nonvolatile memory transformation of client storage", Computer, 46(8), pp.38~44 (2013)
- Z. Guz, "NVMe-over-fabrics performance characterization and the path to low-overhead flash disaggregation", ACM International Systems and Storage Conference (2017)
- E. Dart, "The science DMZ: A network design pattern for data-intensive science", International Conference for High Performance Computing, Networking, Storage and Analysis (2013)
- N. Manchanda, "Non-uniform memory access (numa), Vol. 4" (2010), New York University
- N. Hanford, "Improving network performance on multicore systems : Impact of core affinities on high throughput flows", Future Genereration Computer System, 56, pp.277~283 (2016)
- R. Peinl, "Docker cluster management for the cloud-survey results and own solution", Journal of Grid Computing, 14(2), pp.265~282 (2016)
- 유세영, "Optimizing NUMA System Beyond 100 Gbps for Big Data", 한국지식정보기술학회 논문지, 16(3), pp.393~403 (2021), 한국지식정보기술학회[DOI]
- E. S. Jung, "Toward optimizing disk-to-disk transfer on 100G networks", International Conference on Advanced Networks and Telecommunications Systems (2013)
- A. Rajendran, "Raicu, Optimizing large data transfers over 100gbps wide area networks", International Symposium on Cluster, Cloud, and Grid Computing (2013)
- L. Zhang, "MDTM: Optimizing data transfer using multicore-aware i", o scheduling, 2017, ~
- 김종윤, "Optimized Data Processing Analysis Using Big Data Cloud Platform", 한국지식정보기술학회 논문지, 16(1), pp.1~7 (2021), 한국지식정보기술학회[DOI]
- Z. Guz, "Performance characterization of NVMe-over-fabrics storage disaggregation", Transaction on Storage, 14(4) (2018)
- Z. Majo, "Memory system performance in a NUMA multicore multiprocessor", International Conference on Systems and Storage (2011)
- S. A. Weil, "Ceph: A scalable, high-performance distributed file system", symposium on Operating systems design and implementation (2006)
- P. Schwan, "Lustre: Building a file system for 1000-node clusters", Linux symposium (2003)