Abstract
The recent success of deep learning applications is driven by the computing power of GPUs. However, as the workflow of deep learning becomes increasingly complicated and resource-intensive, how to manage the expensive GPU resources for Machine Learning (ML) workload becomes a critical problem. Existing resource managers mostly only focus on a single specific type of workload, like batch processing or web services, and lacks runtime optimization and application performance awareness. Therefore, we aim to develop a set of runtime dynamic management techniques (including auto-scaling, job preemption, workload-aware scheduling, and elastic GPU sharing) to handle a mixture of ML workloads consisting of modeling, training, and inference jobs. In our previous work, we have implemented these techniques as a set of extended operators on Kubernetes. In this paper, we further extend our approach by introducing a topology-aware scheduling algorithm based on the hypergraph partition problem to minimize the communication cost of distributed training for maximizing the system throughput and minimizing the job completion time. Our evaluations on AWS GPU clusters prove our approach can out-perform the native Kubernetes by 60% system throughput improvement, 70% training time reduction without causing any SLA violations on inference services. Compared to the start-of-the-art topology-aware scheduling algorithm, we shorten the average job completion time by 24–44%.