An Experimental Evaluation of a Unified Checklist for Designing and Reporting Empirical Research in Software Engineering

This article reports on an experimental evaluation of a unified checklist for case study and experimental research. The checklist aims at exhibiting the underlying, shared, structure of observational and experimental research, and is based on several published checklist in software engineering and other disciplines. We asked a small sample of subjects to apply it to two papers, one reporting on a case study and one reporting on an experiment. After this experience, we collected their perceptions of the perceived ease of use and usefulness of the checklist. The outcome reveals that respondents find some of the concepts used in the checklist hard to understand, thus difficult to be used by the subjects (researchers). However they also think a unified checklist like this would be useful for their empirical research. We discuss the findings and their limitations, and infer some implications for improving the checklist.