Workshop on Dependability Benchmarking

Classical features such as raw performance and functionality have long driven the computer industry to improve their products. But now, dependability and maintainability are seen as equally important. While there are relatively straightforward ways to evaluate and compare performance and functionality of different systems or components, the evaluation of dependability and maintainability features is much more difficult. Among the challenges that must be addressed are: incorporating the effects of software failures, characterizing the dependability of opaque off-the-shelf hardware and software components, including the effects of typical maintenance, operational, and configuration management procedures, and accommodating the fact that different application areas have different requirements for the various factors influencing dependability. The goal of the Dependability Benchmarking Workshop is to provide a forum for the computer industry and academia to discuss problems associated with the evaluation and characterization of dependability and maintainability of components and computer systems. The identification of dependability benchmarking measures and the essential technologies for dependability benchmarking, including both experimental measuring and modeling technologies, are central aspects of this large discussion meant to garner ideas on practical and cost-effective ways to evaluate dependability and maintainability features. This workshop is the outcome of the first two years of work of the IFIP 10.4 WG SIG on Dependability Benchmarking (SIGDeB) [SIGDeB02]. That SIG was formed in November 1999 under the IFIP 10.4 WG to promote the research, practice, and adoption of benchmarks for computer-related system dependability. Koopman & Madeira were the founding co-chairs. The SIGDeB charter focusses on four areas in particular: • Exchanging ideas about dependability benchmarking, including researchers and practitioners from universities, industry, and government agencies. • Documenting the state of the art for dependability measurement and benchmarking. • Creating lists of issues that must be resolved to advance dependability benchmarking to a mature science. • Eventually, proposing a mechanism and agenda for a group to propose dependability benchmarks.