The State of the Art in Controlled Experimentation on Testing Techniques and an Evaluation Criteria for Controlled Experiments

The importance of empirical studies in software engineering has been recognized by many researchers, but the number of empirical studies is relatively small compared to other disciplines. The quality of individual studies also has not been evaluated well enough to build up knowledge about empirical studies in that field and to provide guidelines for future studies. To investigate the current state of empirical studies, we perform two surveys: a survey of papers that evaluate empirical studies in software engineering, and a survey of papers that provide experimentation guidelines. Based on results from these surveys and additional evaluation criteria identified, we establish criteria for the evaluation of empirical studies on testing techniques. Using this criteria, we evaluate empirical studies of software testing techniques and expose some problems in performing empirical studies. This report intends to provide insights regarding how empirical studies of testing techniques should be performed.