Empirical investigation of fault predictors in context of class membership probability estimation

In the domain of software fault prediction, class membership probability of a selected classifier and the factors related to its estimation can be considered as necessary information for tester to take informed decisions about software quality issues. The objective of this study is to empirically investigate the class membership probability estimation capability of 15 classifiers/fault predictors on 12 datasets of open source projects retrieved from PROMISE repository. We empirically validate the effect of dataset characteristics and set of metrics on the performance of classifiers in estimating the class membership probability. We used Receiver Operating Characteristics-Area under Curve (ROC-AUC) value and overall accuracy as benchmarks to evaluate and compare the performance of classifiers. We apply Friedman's, post-hoc Nemenyi and Analysis of Means (ANOM) test to compare the significant performance of classifiers. We conclude that ADTree and RandomForest outperform, while ZeroR classifier cannot show significant performance for estimation of class membership probability.