A Scalable Message Passing Interface Implementation of an Ad-Hoc Parallel I/o system
نویسندگان
چکیده
In this paper we present the novel design, implementation, and evaluation of an ad-hoc parallel I/O system (AHPIOS). AHPIOS is the first scalable parallel I/O system completely implemented in the Message Passing Interface (MPI). The MPI implementation brings the advantages of portability, scalability and high performance. AHPIOS allows MPI applications to dynamically manage and scale distributed partitions in a convenient way. The configuration of both the MPIIO and the storage management system is unified and allows for a tight integration of the optimizations of these layers. AHPIOS partitions are elastic: they conveniently scale up and down with the number of resources. We develop two collective I/O strategies, which leverage a two-tiered cooperative cache in order to exploit the spatial locality of dataintensive parallel applications. The file access latency is hidden from the applications through an asynchronous data staging strategy. The two-tiered cooperative cache scales with both the number of processors and storage resources. Our experimental section demonstrates that, with various optimizations, integrated AHPIOS offers a substantial performance benefit over the traditional MPI-IO solutions on both PVFS or Lustre parallel file systems.
منابع مشابه
A scalable MPI implementation of an ad-hoc parallel I/O system
This paper presents the novel design, implementation and evaluation of an ad-hoc parallel I/O system (AHPIOS). AHPIOS is the first scalable parallel I/O system completely implemented in Message Passing Interface (MPI). The MPI implementation brings the advantages of portability, scalability and high-performance. AHPIOS allows MPI applications to dynamically manage and scale distributed partitio...
متن کاملArchitecture, implementation and parallelization of the software to search for periodic gravitational wave signals
The parallelization, design and scalability of the PolGrawAllSky code to search for periodic gravitational waves from rotating neutron stars is discussed. The code is based on an efficient implementation of the F -statistic using the Fast Fourier Transform algorithm. To perform an analysis of data from the advanced LIGO and Virgo gravitational wave detectors’ network, which will start operating...
متن کاملA Domain-Decomposition Message-Passing Approach to Transient Viscous Incompressible Flow using Explicit Time Integration
This paper discusses the design and implementation of solution algorithms suitable for performing transient, incompressible viscous ow simulations on massively parallel computers. The nite element formulation for incompressible ow along with the ad-hoc modi cations for explicit time integration are discussed with an emphasis on implementation aspects for achieving scalable, parallel computation...
متن کاملThe Design and Implementation of a MPI-Based Parallel File System
This paper presents the design of an MPI (Message Passing Interface)-based parallel file system, MPFS. MPI-IO is an extension of MPI which supports flexible logical file partition and physical file organization as well as a rich set of file access functions. MPFS enables users to specify both logical file partitions among user processes and physical file data layouts across data servers. The pr...
متن کاملThe Impact of File Systems on MPI-IO Scalability
As the number of nodes in cluster systems continues to grow, leveraging scalable algorithms in all aspects of such systems becomes key to maintaining performance. While scalable algorithms have been applied successfully in some areas of parallel I/O, many operations are still performed in an uncoordinated manner. In this work we consider, in three file system scenarios, the possibilities for ap...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید
ثبت ناماگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید
ورودعنوان ژورنال:
- IJHPCA
دوره 24 شماره
صفحات -
تاریخ انتشار 2010