Topology-Aware Scheduling on Blue Waters with Proactive Queue Scanning and Migration-Based Job Placement

Modern HPC systems, such as Blue Waters, has a multidimensional torus topology, which makes it hard to achieve both high system utilization and scheduling speed. The low scheduling speed comes from the inefficient resource allocation scheme by using a pre-defined Shape Table, which is highly time consuming. The low system utilization is majorly caused by system fragmentation and node drainage. System fragmentation includes both internal fragmentation due to convex prism shape requirement, and external fragmentation resulted from contiguous allocation strategy. The node drainage is caused by lacking of free contiguous space for allocation upon large job’s schedule. System drainage is due to the insufficient free contiguous space to schedule the incoming large jobs. In this paper, with the objective of resource utilization and scheduling speed improvement, we address the topology-aware scheduling problem on Blue Waters. To improve the scheduling speed, we propose an efficient free partition detection algorithm. To overcome the system drainage by large job, we propose to use proactive queue scanning to search for large job and gain a buffer period to prepare the system for acceptance. With the set of jobs in the scan window and a set of free partition in the system, we can transform the scheduling into off-line placement, which can be modeled as multiple knapsack problem. We design a migration-based job placement heuristic to make space for large job and improve the system utilization. Through extensive simulations of modeled trace data, we demonstrate that our approach works well in terms of improving the system utilization and scheduling speed.