Evaluating the explainability of AI-driven clinical decision support tools
File(s)
Author(s)
Nagendran, Myura
Type
Thesis
Abstract
Most clinical AI will be realised as part of a decision support system rather than an autonomous agent. Keeping the physician in the loop means that evaluating human-AI interaction behaviour is critical to improving adoption and impact at scale, which has hitherto been lacking. Explainable AI (XAI) aims to not only provide recommendations but also to justify the rationale of the AI to clinicians and so achieve its effect via an influence on human decision-making, positively or negatively. Successful deployment of XAI is therefore as much a problem of clinicians’ influenceability, cognition and psychology as one of machine learning. This thesis investigates the impact of XAI on clinical decision-making in intensive care settings, focusing on sepsis management - a domain characterised by uncertainty and practice variation.
Through innovative experiments in low- and high-fidelity settings, this work provides insights into physician-XAI interaction behaviour and shows that AI recommendations have a varied influence on prescription decisions depending on the experimental setup and XAI type. Eye-tracking in high-fidelity simulation settings was feasible and valuable for objectively assessing physician attention and cognitive load. Importantly, a consistent lack of correlation was found between physician self-reported XAI usefulness and its actual influence on decision-making, challenging the reliability of self-reports as a sole metric for evaluating XAI. The final study evaluated large language models for dynamic XAI, facilitating bi-directional influence between physician and AI while providing the potential for enhancing the clinical reasoning of both.
The findings from this thesis have important implications for all healthcare AI stake-holders involved in attempting to bring XAI-driven decision support tools toward real-world clinical implementation. Specifically, this thesis supports an emphasis on high-fidelity simulation, neuro-behavioural objective markers of attention such as real-time eye-tracking and a more nuanced understanding of the interaction behaviour between physicians and AI in high-stakes healthcare settings.
Through innovative experiments in low- and high-fidelity settings, this work provides insights into physician-XAI interaction behaviour and shows that AI recommendations have a varied influence on prescription decisions depending on the experimental setup and XAI type. Eye-tracking in high-fidelity simulation settings was feasible and valuable for objectively assessing physician attention and cognitive load. Importantly, a consistent lack of correlation was found between physician self-reported XAI usefulness and its actual influence on decision-making, challenging the reliability of self-reports as a sole metric for evaluating XAI. The final study evaluated large language models for dynamic XAI, facilitating bi-directional influence between physician and AI while providing the potential for enhancing the clinical reasoning of both.
The findings from this thesis have important implications for all healthcare AI stake-holders involved in attempting to bring XAI-driven decision support tools toward real-world clinical implementation. Specifically, this thesis supports an emphasis on high-fidelity simulation, neuro-behavioural objective markers of attention such as real-time eye-tracking and a more nuanced understanding of the interaction behaviour between physicians and AI in high-stakes healthcare settings.
Version
Open Access
Date Issued
2024-08-04
Date Awarded
2025-04-01
Copyright Statement
Attribution-Non Commercial-No Derivatives 4.0 International Licence (CC BY-NC-ND)
Advisor
Faisal, Aldo
Gordon, Anthony
Sponsor
UK Research and Innovation
Grant Number
EP/S023283/1
Publisher Department
Department of Computing
Publisher Institution
Imperial College London
Qualification Level
Doctoral
Qualification Name
Doctor of Philosophy (PhD)