RLPTO : A reinforcement learning-based performance-time optimized task and resource scheduling mechanism for distributed machine learning

With the wide application of deep learning, amount data required to train learning models is becoming increasingly larger, resulting in an increased training time and higher requirements for computing resources. To improve throughput a distributed system, task scheduling resource are required. This paper proposes combine ARIMA GRU predict future volume. In terms scheduling, multi-priority queues used divide tasks into different according their priorities ensure that high-priority can be completed advance. reinforcement method adopted manage limited The reward function constructed based on resources occupied by task, time, accuracy model. When model tends converge, gradually reduced so they allocated other tasks. results experiments demonstrate RLPTO use more compu-ting nodes when facing with large scale has good scalability. system experiment shows make cluster get largest reward.