Self-Tuning Database Systems

It is common knowledge that the modern world generates and stores information at an increasing rate. A typical example is the emerging field of e-science, where the adoption of high-performance computing has led to the generation of vast volumes of scientific data. The usefulness of the stored information, of course, depends crucially on the ability to query it effectively, and hence on the use of state of the art data management technology. Unfortunately, despite the growing trends of data collection, the adoption of database systems in e-science has been less than spectacular and has often lead to a lot of frustration. One important reason can be traced to the complexity of managing and tuning a database system. Essentially, the performance of a database system depends crucially on its physical schema, that is, the set of physical structures, such as indices and materialized views, that can speed up the execution of queries. In order to be effective, of course, the physical schema has to match the traits of the workload, e.g., it must index more heavily the parts of the database that are frequently referenced in queries. At the same time, the schema is typically constrained by a total disk space budget for its storage. Thus, designing an effective physical schema involves a nontrivial optimization problem: maximize the throughput of the query processor, assuming limited resources for storing the materialized structures. Clearly, a scientist (or, for that matter, any non-expert user) is not likely to possess the expertise that is necessary in order to tackle this challenging problem. Hiring a database administrator is not a particularly attractive solution either, since this increases the total cost of ownership. To address this important issue, we propose to develop a database system that can self-organize its physical configuration without the intervention of a human administrator. The key idea is to augment the system with an on-line tuning module that monitors continuously the incoming workload, gathers statistics on the performance of the system, and reorganizes the physical design periodically in order to maximize query throughput. Of course, realizing this goal involves several challenging problems. More precisely, since tuning happens concurrently with normal database operation, the tuner has to operate with low and controllable overhead so that it does not affect query performance. Moreover, the tuning module has to track continuously the traits of the workload and identify potential physical designs that can improve performance. Even more importantly, the tuner has to determine whether a recent change in the workload constitutes a major shift in the query distribution that justifies the potentially high cost of changing the physical design. These problems stem from the on-line nature of the problem and have not been addressed by previous studies on database tuning. Thus, the proposed research will make significant scientific contributions in the area of self-tuning systems and autonomic computing. Furthermore, it will lower the barrier for the deployment of database systems and hence facilitate the adoption of database technology in several “data-intensive” disciplines. The proposed project forms part of our more general agenda on the development of database systems that target specifically non-expert users, and in particular scientists. We believe that this user base forms a formidable challenge for database researchers and a chance to bring database systems to the “masses”. In this direction, we have been working in parallel on two systems: the Data Ring [1, 2], a peer-to-peer middleware system that supports declarative complex queries over massively distributed data, and Chameleon, a DBMS that allows in-situ querying for data residing in a file system. The notion of self-organization is prevalent in both systems, since the goal is to enable non-expert users to enjoy the benefits of database technology without the inconvenience of system administration.