Parallelization of an Object-Oriented Unstructured Aeroacoustics Solver

A computational aeroacoustics code based on the discontinuous Galerkin method is ported to several parallel platforms using MPI. The discontinuous Galerkin method is a compact high-order method that retains its accuracy and robustness on non-smooth unstructured meshes. In its semi-discrete form, the discontinuous Galerkin method can be combined with explicit time marching methods making it well suited to time accurate computations. The compact nature of the discontinuous Galerkin method also makes it well suited for distributed memory parallel platforms. The original serial code was written using an objectoriented approach and was previously optimized for cache-based machines. The port to parallel platforms was achieved simply by treating partition boundaries as a type of boundary condition. Code modifications were minimal because boundary conditions were abstractions in the original program. Scalability results are presented for the SGI Origin, IBM SP2, and clusters of SGI and Sun workstations. Slightly superlinear speedup is achieved on a fixed-size problem on the Origin, due to cache effects. Key words, discontinuous Galerkin method, object-oriented, unstructured grids, Euler equations, highorder accuracy, superlinear speedup Subject classification. Computer Science 1. Motivation. Computational Aeroacoustics (CAA) involves the direct simulation of sound generation and/or propagation about an aircraft or an aircraft component. To be of practical value in the aircraft design process, these massive computations must be performed quickly, and to do so requires efficient use of parallel computer platforms. CAA methods must provide both temporal and spatial accuracy beyond what the second-order discretizations employed in most other areas of computational aerodynamics are capable of providing. In addition, such methods must be easy to apply to complex geometries without sacrifice of accuracy or robustness. These requirements further complicate the design of the parallel implementation. For instance, traditional high-order finite-difference methods are not compact and the amount of data that must be moved across partition boundaries increases considerably with the order of the method. The requirement for time accuracy means all partitions must be advanced in lock step. Common techniques used in steady calculations, such as lagging some information or communicating only after several iterations, cannot be employed. The discontinuous Calerkin method is a relatively new approach that satisfies the numerical requirements of CAA and the algorithmic requirements of parallel implementation. Discontinuous Galerkin is a compact *This research was supported by the National Aeronautics and Space Administration under NASA Contract No. NAS197046 while {Baggag, ()zturan, Keyes} were in residence at the Institute for Computer Applications in Science and Engineering (ICASE), NASA Langley Research Center, Hampton, VA 23681-2199 tDepartment of Computer Sciences, Purdue University, 1398 Computer Science Building, West-Lafayette, IN 47907-1398, (e-maih baggag_cs, purdue, edu) SAerodynamic and Acoustic Methods Branch, NASA Langley Research Center, Hampton, VA 23681-2199, (e-mail: h. i. atkins@larc, nasa. gov) §Department of Computer Engineering, Bogazici University, Istanbul, Turkey_ (e-mail: [email protected], tr) _ICASE NASA Langley Research Center, Hampton, VA 23681-2199, (e-mail: keyesQicase.edu) and Old Dominion University, Department of Computer Science, Norfolk, VA 23529-0162, (e-mail: keyes©cs.odu.edu). method that can be applied to structured or unstructured grids. Many of the method's accuracy and stability properties have been rigorously proven [1, 2, 3, 4, 5] for arbitrary element shapes, any number of spatial dimensions, and even for nonlinear problems, which lead to a very robust method. It has been demonstrated in mesh refinement studies [6] that the accuracy of this method does not depend upon the smoothness of the mesh. Furthermore, the method requires no special treatment near boundaries, which are problematic for many high-order methods. These features are crucial for the robust treatment of complex geometries. In semi-discrete form, the discontinuous Galerkin method can be combined with explicit time-marching methods, such as Runge-Kutta, to create a method well suited for CAA applications. The method has been criticized for its high storage and high computational requirements; however, a recently developed quadrature-free implementation [6] has greatly ameliorated these concerns. The quadrature-free form of the discontinuous Galerkin method has been implemented and validated [6, 7, 8] in an object-oriented code for the prediction of aeroacoustic scattering from complex configurations. The same compactness that permits the accuracy to be insensitive to mesh smoothness makes the method well suited for implementation in parallel machines. Biswas, Devine, and Flaherty [9] applied a third-order quadrature-based discontinuous method to a scalar wave equation on a NCUBE/2 hypercube platform. They reported a 97.57% parallel efficiency on 256 processors. In this work, the discontinuous code developed by Atkins has been ported to several parallel platforms using MPI. The code solves the unsteady linear Euler equations on an unstructured mesh in two dimensions. A detailed description of the numerical algorithm can be found in reference [6]; however, the code structure has not been previously described. The first section of this article provides a brief description of the numerical method followed by a more detailed description of the code structure and the object-oriented design of the code. The second section describes the parallelization strategy and the modifications to the code to implement that strategy. The third section presents performance results of the code on the Origin2000 and several other computing platforms. 2. Discontinuous Galerkin Method. The discontinuous Galerkin method can be applied to an equation in conservation form (2.1) OU 0--_ + v. ,_(u) = 0 defined on a domain f_ with appropriate boundary conditions. The domain is divided into smaller, nonoverlapping elements f_i that cover the domain, i.e., f_ = O_i. The discontinuous Galerkin method is obtained by applying a traditional Galerkin method to each element [10]. That is, a fiuite-dimensional basis set is selected for each element, the solution in each element is approximated in terms of an expansion in that basis set, and the governing equations are then projected on each member of the basis set and cast in a weak form (2.2) f 0V_ bk---_dl2 f Vbk.ff(V_)d_ + / bkffR(Vi,_).r_ds ---0 where r_ in an outward-pointing, surface-element normal, B { bk, 1 < k < N(p,d) } denotes the basis set of degree p in d space dimensions, g(p,_) (2.3) Vi -E v_,tbt _ U_,