Investigating the Effects of Balanced Training and Testing Datasets on Effort-Aware Fault Prediction Models

Research output: Chapters, Conference Papers, Creative and Literary Works (RGC: 12, 32, 41, 45)32_Refereed conference paper (with ISBN/ISSN)peer-review

19 Scopus Citations
View graph of relations

Author(s)

Related Research Unit(s)

Detail(s)

Original languageEnglish
Title of host publication2016 IEEE 40th Annual Computer Software and Applications Conference (COMPSAC)
PublisherIEEE
Pages154-163
Volume1
ISBN (Electronic)978-1-4673-8845-0
Publication statusPublished - Aug 2016

Publication series

NameIEEE Annual International Computer Software and Applications Conference (COMPSAC)
PublisherIEEE
ISSN (Print)0730-3157

Conference

Title40th IEEE Computer Society International Conference on Computers, Software & Applications, COMPSAC 2016
PlaceUnited States
CityAtlanta
Period10 - 14 June 2016

Abstract

To prioritize software quality assurance efforts, fault prediction models have been proposed to distinguish faulty modules from clean modules. The performances of such models are often biased due to the skewness or class imbalance of the datasets considered. To improve the prediction performance of these models, sampling techniques have been employed to rebalance the distribution of fault-prone and non-fault-prone modules. The effect of these techniques have been evaluated in terms of accuracy/geometric mean/F1-measure in previous studies; however, these measures do not consider the effort needed to fix faults. To empirically investigate the effect of sampling techniques on the performance of software fault prediction models in a more realistic setting, this study employs Norm(Popt), an effort-aware measure that considers the testing effort. We performed two sets of experiments aimed at (1) assessing the effects of sampling techniques on effort-aware models and finding the appropriate class distribution for training datasets (2) investigating the role of balanced training and testing datasets on performance of predictive models. Of the four sampling techniques applied, the over-sampling techniques outperformed the under-sampling techniques with Random Over-sampling performing best with respect to the Norm(Popt) evaluation measure. Also, performance of all the prediction models improved when sampling techniques were applied between the rates of (20-30)% on the training datasets implying that a strictly balanced dataset (50% faulty modules and 50% clean modules) does not result in the best performance for effort-aware models. Our results also indicate that performances of effort-aware models are significantly dependent on the proportions of the two types of the classes in the testing dataset. Models trained on moderately balanced datasets are more likely to withstand fluctuations in performance as the class distribution in the testing data varies.

Research Area(s)

  • class imbalance, sampling techniques, software quality, software fault prediction, empirical study

Citation Format(s)

Investigating the Effects of Balanced Training and Testing Datasets on Effort-Aware Fault Prediction Models. / Bennin, Kwabena Ebo; Keung, Jacky; Monden, Akito; Kamei, Yasutaka; Ubayashi, Naoyasu.

2016 IEEE 40th Annual Computer Software and Applications Conference (COMPSAC). Vol. 1 IEEE, 2016. p. 154-163 (IEEE Annual International Computer Software and Applications Conference (COMPSAC)).

Research output: Chapters, Conference Papers, Creative and Literary Works (RGC: 12, 32, 41, 45)32_Refereed conference paper (with ISBN/ISSN)peer-review