Maitri: Format Independent Data Management for Scientific Data

Today’s scientific applications are very data intensive, and their data management requirements can no longer be met by special-purpose libraries for particular scientific data formats or by traditional database management systems. This paper proposes Maitri, a data-format-independent, loosely-coupled, application-tailorable set of libraries that provides a holistic data management framework for scientists. A scientific programmer can write applications that access scientific data in its native format, while using Maitri’s APIs for buffering, indexing, metadata management, and concurrency control, plus a small amount of format-specific code written by the programmer. Experiments to date with Maitri’s buffer and index managers show that this format independence can be obtained at an acceptable runtime cost.