Improved heuristic job scheduling method to enhance throughput for big data analytics

Data-parallel computing platforms, such as Hadoop and Spark, are deployed in clusters for big data analytics. There is a general tendency that multiple users share the same cluster. The schedule of jobs becomes serious challenge. Over long period past, Shortest-Job-First (SJF) method has been considered optimal solution to minimize average job completion time. However, SJF leads low system throughput case where small number short consume large amount resources. This factor prolongs We propose an improved heuristic scheduling method, called Densest-Job-Set-First (DJSF) method. DJSF schedules by maximizing completed per unit time, aiming decrease Job Completion Time (JCT) improve throughput. perform extensive simulations based on Google cluster data. Compared with decreases JCT 23.19% enhances 42.19%. Tetris, packing improves efficiency 55.4%, so platforms complete more time span.