Estimating Software Reliability Using Inverse Sampling

|This paper addresses one of the perpetual questions faced by software developers, \How much testing is enough testing?". Software testing accounts for a substantial portion of software development costs, but releasing software with unacceptable reliability is also very costly. We begin with a discussion of how classical notions of reliability can be applied to deterministic software and explain that the reliability of a software is as much a function of the way that it is used as of quality of the software. We then illustrate how simple operational pro les can be used to characterize usage patterns. Finally, we show how the reliability requirements can be used to determine how much testing is necessary and whether or not the software is acceptable. Using the method of inverse sampling, also called negative binomial sampling, we develop an e cient statistical procedure for quantifying the reliability of a piece of software. The procedure allows substantial reductions in the average number of executions run over the traditional binomial testing. Other issues such as the calculation of upper con dence bounds for software failure rate under both binomial and negative binomial sampling are also addressed. The results obtained are illustrated numerically and graphically on several cases arising in practice. Some issues for further work, namely the use of sequential testing, the computer implementation of the methods developed and the testing of continuously-run software, are also discussed. Index Terms|Average number of executions, binomial sampling, negative binomial sampling, operational pro le, software failure rate, software reliability, test case, test of hypotheses, trace, upper con dence bound. Balwant Singh is with the Communications Research Laboratory, McMaster University, Hamilton, Ontario, Canada L8S 4K1. e-mail: [email protected]. Rom an Viveros is with the Department of Mathematics and Statistics, McMaster University, Hamilton, Ontario, Canada L8S 4K1. e-mail: [email protected]. David L. Parnas is with the Communications Research Laboratory, McMaster University, Hamilton, Ontario, Canada L8S 4K1. e-mail: [email protected].