The Significant Effects of Data Sampling Approaches on Software Defect Prioritization and Classification

Research output: Conference Papers (RGC: 31A, 31B, 32, 33)31A_Invited conference paper (refereed items)

View graph of relations

Author(s)

Related Research Unit(s)

Detail(s)

Original languageEnglish
StatePublished - Nov 2017

Conference

Title11th International Symposium On Empirical Software Engineering and Measurement, ESEM 2017
Location
PlaceCanada
CityToronto
Period9-10 November 2017

Abstract

Recent studies have shown that performance of defect prediction models can be affected when data sampling approaches are applied to imbalanced training data for building defect prediction models. However, the magnitude (degree and power) of the effect of these sampling methods on the classification and prioritization performances of defect prediction models is still unknown. Goal: To investigate the statistical and practical significance of using resampled data for constructing defect prediction models. Method: We examine the practical effects of six data sampling methods on performances of five defect prediction models. The prediction performances of the models trained on default datasets (no sampling method) are compared with that of the models trained on resampled datasets (application of sampling methods). To decide whether the performance changes are significant or not, robust statistical tests are performed and effect sizes computed. Twenty releases of ten open source projects extracted from the PROMISE repository are considered and evaluated using the AUC, pd, pf and G-mean performance measures. Results: There are statistical significant differences and practical effects on the
classification performance (pd, pf and G-mean) between models trained on resampled datasets and those trained on the default datasets. However, sampling methods have no statistical and practical effects on defect prioritization performance (AUC) with small or no effect values obtained from the models trained on the resampled datasets. Conclusions: Existing sampling methods can properly set the threshold between buggy and clean samples, while they cannot improve the prediction of defect-proneness itself. Sampling methods are highly recommended for defect classification purposes when all faulty modules are to be considered for testing.

Citation Format(s)

The Significant Effects of Data Sampling Approaches on Software Defect Prioritization and Classification. / Bennin, Kwabena; Keung, Jacky; Monden, Akito; Phannachitta, Passakorn; Mensah, Solomon.

2017. 11th International Symposium On Empirical Software Engineering and Measurement, ESEM 2017, Toronto, Canada.

Research output: Conference Papers (RGC: 31A, 31B, 32, 33)31A_Invited conference paper (refereed items)