Exploring the relationship between performance metrics and cost saving potential of defect prediction models

2022 | journal article. A publication with affiliation to the University of Göttingen.

Jump to: Cite & Linked | Documents & Media | Details | Version history

Cite this publication

​Exploring the relationship between performance metrics and cost saving potential of defect prediction models​
Tunkel, S. & Herbold, S. ​ (2022) 
Empirical Software Engineering27(7).​ DOI: https://doi.org/10.1007/s10664-022-10224-4 

Documents & Media

document.pdf3.23 MBAdobe PDF

License

Published Version

Attribution 4.0 CC BY 4.0

Details

Authors
Tunkel, Steffen; Herbold, Steffen 
Abstract
Abstract Context: Performance metrics are a core component of the evaluation of any machine learning model and used to compare models and estimate their usefulness. Recent work started to question the validity of many performance metrics for this purpose in the context of software defect prediction. Objective: Within this study, we explore the relationship between performance metrics and the cost saving potential of defect prediction models. We study whether performance metrics are suitable proxies to evaluate the cost saving capabilities and derive a theory for the relationship between performance metrics and cost saving potential. Methods: We measure performance metrics and cost saving potential in defect prediction experiments. We use a multinomial logit model, decision, and random forest to model the relationship between the metrics and the cost savings. Results: We could not find a stable relationship between cost savings and performance metrics. We attribute the lack of the relationship to the inability of performance metrics to account for the property that a small proportion of very large software artifacts are the main driver of the costs. fact that performance metrics are incapable of accurately considering the costs associate with individual artifacts, which is required due to the exponential distribution of artifact sizes. Conclusion: Any defect prediction study interested in finding the best prediction model, must consider cost savings directly, because no reasonable claims regarding the economic benefits of defect prediction can be made otherwise.
Issue Date
2022
Journal
Empirical Software Engineering 
Organization
Institut für Informatik 
ISSN
1382-3256
eISSN
1573-7616
Language
English

Reference

Citations


Social Media