쿠버네티스에서 분산 학습 작업 성능 향상을 위한오토스케일링 기반 동적 자원 조정 오퍼레이터
Dynamic Resource Adjustment Operator Based on Autoscaling for Improving Distributed Training Job Performance on Kubernetes
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
딥러닝 분산 학습에 사용되는 많은 도구 중 하나는 컨테이너 오케스트레이션 도구인 쿠버네티스에서 실행되는 큐브플로우이다. 그리고 큐브플로우에서 기본적으로 제공하는 오퍼레이터를 사용하여 텐서플로우 학습 작업을 관리할 수 있다. 하지만 파라미터 서버 아키텍처 기반의 딥러닝 분산학습 작업을 고려할 때 기존의 오퍼레이터가 사용하는 스케줄링 정책은 분산학습 작업의 태스크 친화도를 고려하지 않으며 자원을 동적으로 할당하거나 해제하는 기능을 제공하지 않는다. 이는 작업의 완료 시간이 오래 걸리거나 낮은 자원 활용률로 이어질 수 있다. 따라서 본 논문에서는 작업의완료 시간을 단축시키고 자원 활용률을 높이기 위해 딥러닝 분산 학습 작업을 효율적으로 스케줄링하는 새로운 오퍼레이터를 제안한다. 기존 오퍼레이터를 수정하여 새로운 오퍼레이터를 구현하고 성능 평가를 위한 실험을 수행한 결과, 제안한 스케줄링 정책은 평균 작업 완료 시간 감소율을최대 84%, 평균 CPU 활용 증가율을 최대 92%까지 향상시킬 수 있음을 보여준다.
Abstract
One of the many tools used for distributed deep learning training is Kubeflow, which runs on Kubernetes, a container orchestrationtool. TensorFlow jobs can be managed using the existing operator provided by Kubeflow. However, when considering the distributeddeep learning training jobs based on the parameter server architecture, the scheduling policy used by the existing operator does notconsider the task affinity of the distributed training job and does not provide the ability to dynamically allocate or release resources. This can lead to long job completion time and low resource utilization rate. Therefore, in this paper we proposes a new operator thatefficiently schedules distributed deep learning training jobs to minimize the job completion time and increase resource utilization rate. We implemented the new operator by modifying the existing operator and conducted experiments to evaluate its performance. Theexperiment results showed that our scheduling policy improved the average job completion time reduction rate of up to 84% and averageCPU utilization increase rate of up to 92%.
참고문헌 (27)
- T. Ben-Nun, "Demystifying parallel and distributed deep learning: An in-depth concurrency analysis", ACM Computing Surveys (CSUR), 52(4), pp.1~43 (2019)
- P. Goyal, "Accurate, large minibatch sgd: Training imagenet in 1 hour", arXiv preprint arXiv:1706.02677, ~ (2017)
- X. Jia, "Highly scalable deep learning training system with mixed-precision: Training imagenet in four minutes", arXiv preprint arXiv:1807.11205, ~ (2018)
- C. Ying, "Image classification at supercomputer scale", arXiv preprint arXiv:1811.06992, ~ (2018)
- S. Zhou, "Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients", arXiv preprint arXiv:1606.06160, ~ (2016)
- C. J. Shallue, "Measuring the effects of data parallelism on neural network training", arXiv:1811.03600, ~ (2018)
- M. Li, "Parameter server for distributed machine learning", Big Learning NIPS Workshop (2013)
- "Kubeflow 2021"[URL]
- "TensorFlow Operator 2021"[URL]
- S. Li, "Pytorch distributed: Experiences on accelerating data parallel training", arXiv preprint arXiv: 2006.15704, ~ (2020)
- H. Cui, "Geeps: Scalable deep learning on distributed gpus with a gpu-specialized parameter server", Proceedings of the Eleventh European Conference on Computer Systems (2016)
- "Kubernetes 2021"[URL]
- Y. Peng, "Optimus:An efficient dynamic resource scheduler for deep learning clusters", Proceedings of ACM EuroSys (2018)
- "Operator 2021"[URL]
- J. Gu, "Tiresias: A GPU cluster manager for distributed deep learning", Proceedings of USENIX Symposium on Networked Systems Design and Implementation (NSDI) (2019)
- R. Grandl, "Multi-resource packing for cluster schedulers", ACM SIGCOMM Computer Communication Review, 44(4), pp.455~466 (2014)
- Y. Chen, "Convolutional neural network for sentence classification", MS, pp.2015
- K. Simonyan, "Very deep convolutional networks for large-scale image recognition", arXiv preprint arXiv:1409.1556, ~ (2014)
- K. He, "Deep residual learning for image recognition", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)
- S. Xie, "Aggregated residual transformations for deep neural networks", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)
- "Prometheus 2021"[URL]
- "Grafana 2021"[URL]
- J. Geng, "Accelerating distributed machine learning by smart parameter server", Proceedings of 3rd Asia-Pacific Workshop Networking (2019)
- E. Gebremeskel, "Analysis and comparison of distributed training techniques for deep neural networks in a dynamic environment", ~ (2018)
- W. Xiao, "Gandiva: Introspective cluster scheduling for deep learning", 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18) (2018)
- Y. Bao, "Online job scheduling in distributed machine learning clusters", IEEE INFOCOM 2018-IEEE Conference on Computer Communications (2018)
- M. Khalil-Hani, "A-sdlm: an asynchronous stochastic learning algorithm for fast distributed learning", 13th Australasian Symposium on Parallel and Distributed Computing (2015)