Code, test and coverage evolution in mature software systems: changes over the past decade
File(s)main.pdf (1.73 MB)
Accepted version
Author(s)
Thomas, Bailey
Cadar, Cristian
Type
Conference Paper
Abstract
Despite the central role of test suites in the software
development process, there is surprisingly limited information on how code and tests co-evolve to exercise different parts of the codebase.
Ten years ago, the Covrig project examined the code, test, and coverage evolution in six mature open-source projects, spanning a combined development time of twelve years. In this study, we significantly expand the analysis to nine mature projects and a combined period of 78 years of development time. Our focus is on understanding how development practices have changed and how these changes have impacted the way in which software is tested.
We report on the co-evolution of code and tests; the adoption of CI, coverage, and fuzzing services; the changes to the overall code coverage achieved by developer test suites; the distribution of patch coverage across revisions; how different kinds of code changes impact coverage; and the occurrence and evolution of
flaky tests.
Our large-scale study paints a mixed picture in terms of how software development and testing have changed over the past ten years. While developers put more emphasis on software testing and the overall code coverage achieved by developer test suites has increased in most projects, coverage and fuzzing services are not widely adopted, many patches are still poorly tested, and the fraction of flaky tests has increased.
development process, there is surprisingly limited information on how code and tests co-evolve to exercise different parts of the codebase.
Ten years ago, the Covrig project examined the code, test, and coverage evolution in six mature open-source projects, spanning a combined development time of twelve years. In this study, we significantly expand the analysis to nine mature projects and a combined period of 78 years of development time. Our focus is on understanding how development practices have changed and how these changes have impacted the way in which software is tested.
We report on the co-evolution of code and tests; the adoption of CI, coverage, and fuzzing services; the changes to the overall code coverage achieved by developer test suites; the distribution of patch coverage across revisions; how different kinds of code changes impact coverage; and the occurrence and evolution of
flaky tests.
Our large-scale study paints a mixed picture in terms of how software development and testing have changed over the past ten years. While developers put more emphasis on software testing and the overall code coverage achieved by developer test suites has increased in most projects, coverage and fuzzing services are not widely adopted, many patches are still poorly tested, and the fraction of flaky tests has increased.
Date Acceptance
2025-12-18
Publisher
IEEE
Copyright Statement
Subject to copyright. This paper is embargoed until publication. Once published the author’s accepted manuscript will be made available under a CC-BY License in accordance with Imperial’s Research Publications Open Access policy (www.imperial.ac.uk/oa-policy).
License URL
Sponsor
European Research Council (ERC)
Grant Number
819141
Source
IEEE International Conference on Software Testing, Verification and Validation (ICST 2025)
Publication Status
Accepted
Start Date
2025-03-31
Finish Date
2025-04-04
Coverage Spatial
Naples, Italy