Load Balancing using Hilbert Space-filling Curves for Parallel Reservoir Simulations

New reservoir simulators designed for parallel computers enable us to overcome performance limitations of personal computers and to simulate large-scale reservoir models. With development of parallel reservoir simulators, more complex physics and detailed models can be studied. The key to design efficient parallel reservoir simulators is not to improve the performance of individual CPUs drastically but to utilize the aggregation of computing power of all requested nodes through high speed networks [38]. An ideal scenario is that when the number of MPI (Message Passing Interface) processes is doubled, the running time of parallel reservoir simulators is reduced by half. The goal of load balancing (grid partitioning) is to minimize overall computations and communications, and to make sure that all processors have a similar workload [4]. Geometric methods divide a grid by using a location of a cell [8] while topological methods work with connectivity of cells, which is generally described as a graph [20]. This paper introduces a Hilbert space-filling curve method. A space-filling curve is a continuous curve and defines a map between a one-dimensional space and a multi-dimensional space [25]. A Hilbert space-filling curve is one special space-filling curve discovered by Hilbert and has many useful characteristics [25, 26], such as good locality, which means that two objects that are close to each other in a multi-dimensional space are also close to each other in a one dimensional space. This property can model communications in grid-based parallel applications. The idea of the Hilbert space-filling curve method is to map a computational domain into a one-dimensional space, partition the one-dimensional space to certain intervals, and assign all cells in a same interval to a MPI process. To implement a load balancing method, a mapping kernel is required to convert high-dimensional coordinates to a scalar value and an efficient one-dimensional partitioning module that divides a one-dimensional space and makes sure that all intervals have a similar workload. The Hilbert space-filling curve method is compared with ParMETIS, a famous graph partitioning package. The results show that our Hilbert space-filling curve method has good partition quality. It has been applied to grids with billions of cells, and linear scalability has been obtained on IBM Blue Gene/Q.