PetaScale Calculations of the Electronic Structures of Nanostructures with Hundreds of Thousands of Processors

OVERVIEW: Density functional theory (DFT) is the most widely used ab initio method in material simulations. It accounts for 75% of the NERSC allocation time in the material science category. The DFT can be used to calculate the electronic structure, the charge density, the total energy and the atomic forces of a material system. With the advance of the HPC power and new algorithms, DFT can now be used to study thousand atom systems in some limited ways (e.g, a single selfconsistent calculation without atomic relaxation). But there are many problems which either requires much larger systems (e.g, >100,000 atoms), or many total energy calculation steps (e.g. for molecular dynamics or atomic relaxations). Examples include: grain boundary, dislocation energies and atomic structures, impurity transport and clustering in semiconductors, nanostructure growth, electronic structures of nanostructures and their internal electric fields. Due to the O(N) scaling of the conventional DFT algorithms (as implemented in codes like Qbox, Paratec, Petots), these problems are beyond the reach even for petascale computers. As the proposed petascale computers might have millions of processors, new computational paradigms and algorithms are needed to solve the above large scale problems. In particular, O(N) scaling algorithms with parallelization capability up to millions of processors are needed. For a large material science problem, a natural approach to achieve this goal is by divide-and-conquer method: to spatially divide the system into many small pieces, and solve each piece by a small local group of processors. This solves the O(N) scaling and the parallelization problem at the same time. However, the challenge of this approach is for how to divide the system into small pieces and how to patch them up without the trace of the spatial division. Here, we present a linear scaling 3 dimensional fragment (LS3DF) method which uses a novel division-patching scheme that cancels out the artificial boundary effects of the spatial division. As a result, the LS3DF results are essential the same as the original full system DFT results (with the difference smaller than chemical accuracy and smaller than other numerical uncertainties, e.g, due to numerical grids), while with a required floating point operation thousands of times smaller, and computational time thousands of times shorter, than the conventional DFT method. For example, using a few thousand processors, the LS3DF can calculate a >10,000 atom system within an hour while the conventional method might take more than a month to finish. The LS3DF method is applicable to insulator and semiconductor systems, it covers a current gap in DOE’s material science code portfolio for ab initio ultrascale simulation. We will use it here to solve the internal electric field problems for composite nanostructures.