Dependability Benchmarking of Linux based Systems

Known benchmarks, like SPECweb, SPECint or TPC, evaluate system performance. In general, they evaluate characteristics like speed, response time, usage of resources etc. They usually ignore issues of reliability and dependability. This paper shows a way to extend these performance benchmarks by dependability aspects. It addresses the issues: How does system performance change facing hardware faults? How can we evaluate measures for these performance changes? An example of a dependability benchmark for database systems exercises the methodology of extending a well known benchmark, the TPC BenchmarkC, to a dependability benchmark for online transaction processing. At last the implementation of that dependability benchmark shows the operational accuracy of the approach. FAUmachine, a fast PC-hardware emulator with the ability of fault injection, supported by a faultload generator is used to stress an Oracle database server running on that virtual Linux system. The results are performance measures showing the loss of performance in presence of hardware faults.