Artificial Intelligence based framework for efficient management of resources in data centers
1
조회
0
다운로드
3
인용
0
참고문헌
키워드
초록
Recent technological advancement has enabled the creation of large scale data (big data) in different domains which have raised the need for large data centers to process and analyze that big data in an efficient fashion. The incremental amount of data has made job scheduling and resource management an essential and challenging component of modern-day data centers. Therefore, in this paper, we have proposed a novel artificial intelligence based framework to enhance the data center performance by improving resource management and job scheduling mechanisms. We have used machine learning to predict resource utilization based on user demand over a particular time span for resource management and improved ant colony algorithm to job scheduling. Results generated from framework simulation demonstrates that predicting future demands aids prediction of future resource utilization and enables efficient job scheduling that reduce the number of running nodes by 10% and increase throughput by 15% from traditional resource management and job scheduling methods. Based on our simulation results be to argue that by using artificial intelligence and machine learning we can improve overall performance or modern-day data centers by efficient resource allocation and job scheduling.
Abstract
Recent technological advancement has enabled the creation of large scale data (big data) in different domains which have raised the need for large data centers to process and analyze that big data in an efficient fashion. The incremental amount of data has made job scheduling and resource management an essential and challenging component of modern-day data centers. Therefore, in this paper, we have proposed a novel artificial intelligence based framework to enhance the data center performance by improving resource management and job scheduling mechanisms. We have used machine learning to predict resource utilization based on user demand over a particular time span for resource management and improved ant colony algorithm to job scheduling. Results generated from framework simulation demonstrates that predicting future demands aids prediction of future resource utilization and enables efficient job scheduling that reduce the number of running nodes by 10% and increase throughput by 15% from traditional resource management and job scheduling methods. Based on our simulation results be to argue that by using artificial intelligence and machine learning we can improve overall performance or modern-day data centers by efficient resource allocation and job scheduling.
참고문헌 (19)
- S. Erevelles, "Big data consumer analytics and the transformation of marketing", Journal of Business Research, 69(2), pp.897~904 (2016)
- L. Bondan, "Fende: Marketplace-based distribution, execution, and life cycle management of vnfs", IEEE Communications Magazine, 57(1), pp.13~19 (2019)
- M. Demirci, "A survey of machine learning applications for energy-efficient resource management in cloud computing environments", Machine Learning and Applications (ICMLA), 2015 IEEE 14th International Conference on, pp.1185~1190 (2015), IEEE
- L. Liu, "Exploring customizable heterogeneous power distribution and management for datacenter", IEEE Transactions on Parallel and Distributed Systems (2018)
- A. -H. Fawaz, "Dynamic allocation of power delivery paths in consolidated data centers based on adaptive ups switching", Computer Networks, 144, pp.254~270 (2018)
- C. Li, "Oasis : Scaling out datacenter sustainably and economically", IEEE Transac-tions on Parallel and Distributed Systems, 28(7), pp.1960~1973 (2017)
- N. J. Yadwadkar, "Machine Learning for Automatic Resource Management in the Datacenter and the Cloud", PhD, pp.2018
- D. Cheng, "Heterogeneity aware workload management in distributed sustainable datacenters", IEEE Transactions on Parallel and Distributed Systems, 30(2), pp.375~387 (2019)
- K. Hadjar, "A new approach for scheduling tasks and", or jobs in big data cluster, 2019 (2019), 1~4[DOI]
- K. Hazelwood, "Applied machine learning at facebook: A datacenter infrastructure perspective", 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA), pp.620~629 (2018), IEEE
- R. Yang, "Intelligent resource scheduling at scale:a machine learning perspective", 2018 IEEE Symposium on Service-Oriented System Engineering (SOSE), pp.132~141 (2018), IEEE
- F. M. Cornejo, "Job schedulers for machine learning and data mining algo- rithms distributed in hadoop", VI Jornadas de Cloud Computing & Big Data (JCC&BD)(La Plata, 2018) (2018)
- B. Wang, "Modelling and developing conflict-aware scheduling on largescale data centres", Future Generation Computer Systems, 86, pp.995~1007 (2018)
- C. Witt, "Predictive performance modeling for distributed computing using black-box monitoring and machine learning", arXiv preprint arXiv:1805.11877, ~ (2018)
- Alanazi Rayan, "Resource Prediction for Big Data Processing in a Cloud Data Center: A Machine Learning Approach", IEIE Transactions on Smart Processing & Computing, 7(6), pp.478~488 (2018), 대한전자공학회[DOI]
- J. L. Berral, "Towards energy-aware scheduling in data centers using machine learning", Proceedings of the 1st International Conference on energy-Efficient Computing and Networking, pp.215~224 (2010), ACM
- M. Duggan, "A reinforcement learning approach for dynamic selection of virtual machines in cloud data centres", Innovative Computing Technology (INTECH), 2016 Sixth International Conference on, pp.92~97 (2016), IEEE
- A. Rayan, "Energy-aware resource predic- tion in virtualized data centers: A machine learning approach", Consumer Electronics-Asia (ICCE-Asia), IEEE International Conference on, pp.1~6 (2018), IEEE
- D. Choi, "An enhanced data-locality-aware task scheduling algorithm for hadoop applications", IEEE Systems Journal, 1(99), pp.1~12 (2017)