Efficient Scheduling for Inference of Complex Neural Network Models on Multi-GPU Systems
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
The main challenge facing recent complex neural network models, which have shown competitive accuracy, is their efficient deploymentin multi-GPU systems. The complex inter-layer dependences of the neural network models combined with the variable data communicationoverhead of multi-GPU systems make it almost impossible to achieve a fair performance gain under manual scheduling. To address thisproblem, we propose a new layer-scheduling approach called NN Maestro, which generates an efficient parallel execution strategy formulti-GPU systems that minimizes the data communication overhead, thereby improving the inference latency for complex neural networkmodels. NN Maestro evaluates the advantages of multi-GPU scheduling using a pre-trained SVM classifier and calculates the schedulingorder of layers based on Topological Sort and Significance Cost. Then, NN Maestro selects the optimal GPU by comparing PlacementCosts and generates the final scheduling result by grouping layers for parallel execution. On various multi-GPU configurations (2 2080Ti,4 V100, and 4 A6000 GPUs), NN Maestro achieves up to 1.67x of performance improvement over the baseline.
Abstract
The main challenge facing recent complex neural network models, which have shown competitive accuracy, is their efficient deploymentin multi-GPU systems. The complex inter-layer dependences of the neural network models combined with the variable data communicationoverhead of multi-GPU systems make it almost impossible to achieve a fair performance gain under manual scheduling. To address thisproblem, we propose a new layer-scheduling approach called NN Maestro, which generates an efficient parallel execution strategy formulti-GPU systems that minimizes the data communication overhead, thereby improving the inference latency for complex neural networkmodels. NN Maestro evaluates the advantages of multi-GPU scheduling using a pre-trained SVM classifier and calculates the schedulingorder of layers based on Topological Sort and Significance Cost. Then, NN Maestro selects the optimal GPU by comparing PlacementCosts and generates the final scheduling result by grouping layers for parallel execution. On various multi-GPU configurations (2 2080Ti,4 V100, and 4 A6000 GPUs), NN Maestro achieves up to 1.67x of performance improvement over the baseline.
참고문헌 (32)
- R. Albert, "Statistical Mechanics of Complex Networks", Reviews of Modern Physics, 74(1), pp.47~97 (2002)
- B. E. Boser, "A Training Algorithm for Optimal Margin Classifiers", Proceedings of the Fifth Annual Workshop on Computational Learning Theory (COLT ’92), pp.144~152 (1992), Association for Computing Machinery
- C. C. Chang, "LIBSVM : A Library for Support Vector Machines", ACM Transactions on Intelligent Systems and Technology, 2(3), pp.1~27 (2011)
- T. Chen, "TVM : An Automated End-to-End Optimizing Compiler for Deep Learning", 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’18), pp.578~594 (2018)
- H. P. Cheng, "SwiftNet: Using Graph Propagation as Meta-knowledge to Search Highly Representative Neural Architectures", ~ (2019)
- "NVIDIA DGX-1 with Tesla V100System Architecture"[URL]
- "NVIDIA Telsa V100 Architecture"[URL]
- "NVIDIA Turing GPU Architecture"[URL]
- "NVIDIA Ampere GA 102 GPU Architecture"[URL]
- C. Cortes, "Support-Vector Networks", Journal of Machine Learning, 20(3), pp.273~297 (1995)
- P. Erdős, "On The Evolution of Random Graphs", pp.17~61 (1960), the Mathematical Institute of the Hungarian Academy of Sciences
- K. He, "Deep Residual Learning for Image Recognition", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.770~778 (2016)
- M. A. Iverson, "Parallelizing Existing Applications in a Distributed Heterogeneous Environment", 4th Heterogeneous Computing Workshop (HCW ’95), pp.93~100 (1995)
- Z. Jia, "Beyond Data and Model Parallelism for Deep Neural Networks", Proceedings of Machine Learning and Systems, 1, pp.1~13 (2019)
- S. Kang, "PreScaler : An Efficient System-Aware Precision Scaling Framework on Heterogeneous Systems", Proceedings of the 25th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’20), pp.280~292 (2020)
- A. Krizhevsky, "Learning Multiple Layers of Features from Tiny Images", ~ (2009)
- W. Kwon, "Nimble : Light weight and Parallel GPU Task Scheduling for Deep Learning", Advances in Neural Information Processing Systems, 33, pp.8343~8354 (2020)
- H. Liu, "DARTS : Differentiable Architecture Search", International Conference on Learning Representations (ICLR ’18) (2018)
- R. Mayer, "The Tensorflow Partitioning and Scheduling Problem: It’s the Critical Path!", Proceedings of the 1st Workshop on Distributed Infrastructures for Deep Learning, pp.1~6 (2017)
- Y. Park, "SIMD Defragmenter : Efficient ILP Realization on Data-Parallel Architectures", Proceedings of the 17th International Conference on Architectural Support for Programming Languages and Operating Systems, pp.363~374 (2012)
- A. Paszke, "PyTorch : An Imperative Style, High Performance Deep Learning Library", Advances in Neural Information Processing Systems, 32, pp.8026~8037 (2019)
- F. Pedregosa, "Scikit-learn: Machine Learning in Python", Journal of Machine Learning Research, 12, pp.2825~2830 (2011)
- J. C. Platt, "Advances in Large Margin Classifiers", pp.61~74 (1999), MIT Press
- E. Real, "Regularized Evolution for Image Classifier Architecture Search", Proceedings of the AAAI Conference on Artificial Intelligence, 33, pp.4780~4789 (2019)
- C. Szegedy, "Going Deeper with Convolutions", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.1~9 (2015)
- H. Topcuoglu, "Performance Effective and Low-Complexity Task Scheduling for Heterogeneous Computing", IEEE Transactions on Parallel and Distributed Systems, 13, pp.260~274 (2002)
- H. Wang, "Swiftnet : Real-Time Video Object Segmentation", Proceedings of the IEEE, 2021, 1296~1305
- D. J. Watts, "Collective Dynamics of ‘Small-World’ Networks", Nature, 393, pp.440~442 (1998)
- S. Xie, "Exploring Randomly Wired Neural Networks for Image Recognition", Proceedings of the IEEE, 2019, 1284~1293
- B. Zoph, "Neural Architecture Search with Reinforcement Learning", ~ (2016)
- B. Zoph, "Learning Transferable Architectures for Scalable Image Recognition", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.8697~8710 (2018)