An evaluation of pure spectrum-based fault localization techniques for large-scale software systems
File(s)
Accepted version
Author(s)
Type
Journal Article
Abstract
Pure spectrum-based fault localization (SBFL) is a well-studied statistical debugging technique that only takes a set of test cases (some failing and some passing) and their code coverage as input and produces a ranked list of suspicious program elements to help the developer identify the location of a bug that causes a failed test case. Studies show that pure SBFL techniques produce good ranked lists for small programs. However, our previous study based on the iBugs benchmark that uses the AspectJ repository shows that, for realistic programs, the accuracy of the ranked list is not suitable for human developers. In this paper, we confirm this based on a combined empirical evaluation with the iBugs and the Defects4J benchmark. Our experiments show that, on average, at most ∼40%, ∼80%, and ∼90% of the bugs can be localized reliably within the first 10, 100, and 1000 ranked lines, respectively, in the Defects4J benchmark. To reliably localize 90% of the bugs with the best performing SBFL metric D∗, ∼450 lines have to be inspected by the developer. For human developers, this remains unsuitable, although the results improve compared with the results for the AspectJ benchmark. Based on this study, we can clearly see the need to go beyond pure SBFL and take other information, such as information from the bug report or from version history of the code lines, into consideration.
Date Issued
2019-08-01
Date Acceptance
2019-03-28
Citation
Software - Practice and Experience, 2019, 49 (8), pp.1197-1224
ISSN
0038-0644
Publisher
Wiley
Start Page
1197
End Page
1224
Journal / Book Title
Software - Practice and Experience
Volume
49
Issue
8
Copyright Statement
© 2019 Owner. This is the accepted version of the following article: Heiden, S, Grunske, L, Kehrer, T, et al. An evaluation of pure spectrum‐based fault localization techniques for large‐scale software systems. Softw: Pract Exper. 2019; 1– 28. https://doi.org/10.1002/spe.2703, which has been published in final form at https://doi.org/10.1002/spe.2703.
Subjects
Science & Technology
Technology
Computer Science, Software Engineering
Computer Science
debugging
empirical studies
fault localization
SLICE
Software Engineering
08 Information and Computing Sciences
17 Psychology and Cognitive Sciences
Publication Status
Published
Date Publish Online
2019-05-24