Effectiveness of TDD on Unit Testing Practice

Background: Several studies report that test driven development (TDD) has effects on the software product e.g. code quality and developers’ productivity. In recent literature reviews, the impact of TDD on source code quality is seen as a more focused area in empirical research compared to unit testing. However, the quality of production code is tightly coupled with the quality of test code. Objective: The aim of this study is to investigate the impact of test driven development approach compared to an incremental test last development (ITLD), on unit test case quality. The impact is measured in terms of code coverage and mutation score metrics. The hypotheses test the differences in the quality of test cases produced using TDD and ITLD approaches. Method: We conducted an experiment in an industrial setting with 24 professionals in five consecutive days. Three programing tasks, i.e. one task using ITLD and two tasks using TDD, are selected for the experiment. We extracted unit test case quality attributes i.e. mutation score and code coverage from the data collected on each day of the experiment. For the code coverage, we used the metrics; instruction, branch, method, cyclomatic complexity and line coverage. The difference of mutation score and code coverage metrics are then evaluated using non-parametric significance tests. Results: The results indicate that except three metrics i.e. branch coverage, method coverage, mutation score, we could not find significant differences in terms of unit test case quality between the treatments. Subjects wrote test cases that cover more branches during TDD practice on a green-field (as a new development) task, compared to ITLD and TDD practice on a brown-field (modifications on an existing code base) task. In terms of method coverage, test cases produced using ITLD covered more methods than both TDD tasks; whereas no significant difference is found for method coverage between the both TDD tasks. In terms of mutation score, test cases written during TDD practice on a green-field task have more defect detection abilities than test cases written during ITLD practice and TDD practice on a brown-field task. Conclusion: Our finding are different from previous studies performed at academic settings. We believe that other factors (i.e. task’s complexity, experiment duration and participant’s interest towards tasks) could influence the results. Therefore, future studies could be designed by minimizing the impact of these factors to get more generalizable results.