The impact of using biased performance metrics on software defect prediction research

Software engineering researchers have undertaken many experiments investigating the potential of software defect prediction algorithms. Unfortunately some widely used performance metrics are known to be problematic, most notably F1, but nevertheless F1 is used. To investigate impact using on validity this large body research. We undertook a systematic review locate relevant and then extract all pairwise comparisons unbiased Matthews correlation coefficient (MCC). found total 38 primary studies. These contain 12,471 pairs results. Of these comparisons, 21.95% changed direction when MCC metric instead biased metric. Unfortunately, we also evidence suggesting that remains in reiterate concerns statisticians problematic outside an information retrieval context, since concerned about both classes (defect-prone not defect-prone units). This inappropriate usage has led substantial number (more than one fifth) erroneous (in terms direction) Therefore urge (i) use (ii) publish detailed results including confusion matrices such alternative analyses become possible.