Code, test and coverage evolution in mature software systems: changes over the past decade
File(s)
Accepted version
Author(s)
Thomas, Bailey
Cadar, Cristian
Type
Conference Paper
Abstract
Despite the central role of test suites in the software development process, there is surprisingly limited information on how code and tests co-evolve to exercise different parts of the codebase. A decade ago, the Covrig project examined the code, test, and coverage evolution in six mature open-source C/C++ projects, spanning a combined development time of twelve years. In this study, we significantly expand the analysis to nine mature C/C++ projects and a combined period of 78 years of development time. Our focus is on understanding how development practices have changed and how these changes have impacted the way in which software is tested. We report on the co-evolution of code and tests; the adoption of CI, coverage, and fuzzing services; the changes to the overall code coverage achieved by developer test suites; the distribution of patch coverage across revisions; how code changes drive changes in coverage; and the occurrence and evolution of flaky tests. Our large-scale study paints a mixed picture in terms of how software development and testing have changed over the past decade. While developers put more emphasis on software testing and the overall code coverage achieved by developer test suites has increased in most projects, coverage and fuzzing services are not widely adopted, many patches are still poorly tested, and the fraction of flaky tests has increased.
Date Issued
2025-05-20
Date Acceptance
2024-12-18
Citation
2025 IEEE Conference on Software Testing, Verification and Validation (ICST), 2025, pp.210-220
ISSN
2159-4848
Publisher
IEEE
Start Page
210
End Page
220
Journal / Book Title
2025 IEEE Conference on Software Testing, Verification and Validation (ICST)
Copyright Statement
Copyright © 2025, IEEE. This is the author’s accepted manuscript made available under a CC-BY licence in accordance with Imperial’s Research Publications Open Access policy (www.imperial.ac.uk/oa-policy)
License URL
Sponsor
European Research Council (ERC)
Grant Number
819141
Source
IEEE International Conference on Software Testing, Verification and Validation (ICST 2025)
Publication Status
Published
Start Date
2025-03-31
Finish Date
2025-04-04
Coverage Spatial
Naples, Italy