Safety of human-AI cooperative decision-making within intensive care: a physical simulation study
File(s)
Published version
Author(s)
Festor, Paul
Nagendran, Myura
Gordon, Anthony C
Faisal, Aldo A
Komorowski, Matthieu
Type
Journal Article
Abstract
The safety of Artificial Intelligence (AI) systems is as much one of human decision-making as a technological question. In AI-driven decision support systems, particularly in high-stakes settings such as healthcare, ensuring the safety of human-AI interactions is paramount, given the potential risks of following erroneous AI recommendations. To explore this question, we ran a safety-focused clinician-AI interaction study in a physical simulation suite. Physicians were placed in a simulated intensive care ward, with a human nurse (played by an experimenter), an ICU data chart, a high-fidelity patient mannequin and an AI recommender system on a display. Clinicians were asked to prescribe two drugs for the simulated patients suffering from sepsis and wore eye-tracking glasses to allow us to assess where their gaze was directed. We recorded clinician treatment plans before and after they saw the AI treatment recommendations, which could be either ‘safe’ or ‘unsafe’. 92% of clinicians rejected unsafe AI recommendations vs 29% of safe ones. Physicians paid increased attention (+37% gaze fixations) to unsafe AI recommendations vs safe ones. However, visual attention on AI explanations was not greater in unsafe scenarios. Similarly, clinical information (patient monitor, patient chart) did not receive more attention after an unsafe versus safe AI reveal suggesting that the physicians did not look back to these sources of information to investigate why the AI suggestion might be unsafe. Physicians were only successfully persuaded to change their dose by scripted comments from the bedside nurse 5% of the time. Our study emphasises the importance of human oversight in safety-critical AI and the value of evaluating human-AI systems in high-fidelity settings that more closely resemble real world practice.
Editor(s)
Frasch,, Martin G
Date Issued
2025-02-24
Date Acceptance
2024-12-17
Citation
PLOS Digital Health, 2025, 4 (2)
ISSN
2767-3170
Publisher
Public Library of Science (PLoS)
Start Page
e0000726
End Page
e0000726
Journal / Book Title
PLOS Digital Health
Volume
4
Issue
2
Copyright Statement
© 2025 Festor et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
License URL
Publication Status
Published
Article Number
ARTN e0000726
Date Publish Online
2025-02-24