High-coverage metamorphic testing of concurrency support in C compilers
Author(s)
Windsor, Matt
Donaldson, Alastair F
Wickerson, John
Type
Journal Article
Abstract
We present a technique and automated toolbox for randomized testing of C compilers. Unlike prior compiler-testing approaches, we generate concurrent test cases in which threads communicate using fine-grained atomic operations, and we study actual compiler implementations rather than abstract mappings. Our approach is (1) to generate test cases with precise oracles directly from an axiomatization of the C concurrency model; (2) to apply metamorphic fuzzing to each test case, aiming to amplify the coverage they are likely to achieve on compiler codebases; and (3) to execute each fuzzed test case extensively on a range of real machines. Our tool, C4, benefits compiler developers in two ways. First, test cases generated by C4 can achieve line coverage of parts of the LLVM C compiler that are reached by neither the LLVM test suite nor an existing (sequential) C fuzzer. This information can be used to guide further development of the LLVM test suite and can also shed light on where and how concurrency-related compiler optimizations are implemented. Second, C4 can be used to gain confidence that a compiler implements concurrency correctly. As evidence of this, we show that C4 achieves high strong mutation coverage with respect to a set of concurrency-related mutants derived from a recent version of LLVM and that it can find historic concurrency-related bugs in GCC. As a by-product of concurrency-focused testing, C4 also revealed two previously unknown sequential compiler bugs in recent versions of GCC and the IBM XL compiler.
Date Issued
2022-06
Date Acceptance
2022-02-07
Citation
Software Testing, Verification and Reliability, 2022, 32 (4)
ISSN
0960-0833
Publisher
Wiley
Journal / Book Title
Software Testing, Verification and Reliability
Volume
32
Issue
4
Copyright Statement
© 2022 The Authors. Software Testing, Verification & Reliability published by John Wiley & Sons Ltd.
This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.
This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.
License URL
Identifier
https://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=PARTNER_APP&SrcAuth=LinksAMR&KeyUT=WOS:000771783300001&DestLinkType=FullRecord&DestApp=ALL_WOS&UsrCustomerID=a2bf6146997ec60c407a63945d4e92bb
Subjects
atomics
C11
Computer Science
Computer Science, Software Engineering
concurrency
CONSISTENCY
CORRECT
fuzz testing
program generation
Science & Technology
Technology
VERIFICATION
weak memory
WEAK MEMORY
Publication Status
Published
Article Number
e1812
Date Publish Online
2022-03-22