Unsound foundations: refining AI’s role in audio-based COVID-19 detection

Abstract

This thesis explores the potential for respiratory audio biomarkers to enable detection of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection, with a focus on discerning the true efficacy of Artificial Intelligence (AI) classifiers in this context. The research was motivated by the early pandemic findings suggesting potential vocal biomarkers in COVID-19 infected individuals and the subsequent surge in studies utilising Machine Learning (ML) methodologies for infection detection. Early results raised concerns about whether the success of the ML models was due to actual respiratory audio biomarkers or to confounding signals in the audio recordings, such as recruitment bias. To address these concerns, a large-scale study was undertaken as part of the UK government’s pandemic response, leading to the collection of The UK COVID Vocal Audio Dataset, comprising 72,999 individuals, including 23,514 with positive reverse transcription polymerase chain reaction (PCR) tests for SARS-CoV-2. Unadjusted analysis of this dataset indicated high accuracy (Receiver Operating Characteristic Area Under the Curve (ROC-AUC)=0.846) of AI classifiers in predicting SARS-CoV-2 infection. However, adjusting for measured confounders, such as self-reported symptoms, revealed significantly weaker performance (ROC-AUC=0.619) with a further drop to random classification performance when cases hypothesised to contain unmeasured confounders were removed. A key finding was that, in practical settings, AI classifiers, while initially promising, were outperformed by predictions based on user-reported symptoms. The thesis concludes with the release of the collected dataset and offers best-practice recommendations for future studies, particularly in the handling of recruitment bias and assessment of audio-based classifiers’ utility in relevant practical contexts. This research provides valuable insights into the capabilities and limitations of AI in medical diagnostics and across the wider field of applications, emphasising the importance of careful study design and consideration of confounders in AI-enabled diagnostic tools.