Clinical utility and functionality of an artificial intelligence application to predict mortality in COVID-19: a mixed methods analysis.

File Description SizeFormat 
PDF (7).pdfPublished version647.76 kBAdobe PDFView/Open
Title: Clinical utility and functionality of an artificial intelligence application to predict mortality in COVID-19: a mixed methods analysis.
Authors: Abdulaal, A
Patel, A
Al-Hindawi, A
Charani, E
Alqahtani, S
Davies, G
Mughal, N
Moore, L
Item Type: Journal Article
Abstract: Background The artificial neural network (ANN) is an increasingly important tool in the context of solving complex medical classification problems. However, one of the principal challenges in leveraging AI technology in the healthcare setting has been the relative inability to translate models into clinician workflow. Here we demonstrate the development of a COVID-19 outcome prediction application which utilises an ANN and assesses its usability in the clinical setting. Methods Usability assessment was conducted on the application followed by a semi-structured end-user interview. Usability was specified by effectiveness, efficiency, and satisfaction measures. These data were reported with descriptive statistics. The end-user interview data were analysed using the thematic framework method, which allowed for the development of themes from the interview narratives. Participants Thirty-one Nation Health Service (NHS) physicians at a West London teaching hospital, including foundation doctors, senior house officers, registrars, and consultants. Results All participants were able to complete the assessment, with a mean time to complete separate patient vignettes of 59.35 seconds (standard deviation (SD) = 10.35). Mean system usability scale (SUS) score was 91.94 (SD = 8.54), which corresponds with an adjective rating of “Excellent”. The clinicians found the application intuitive and easy to use, with the majority describing its predictions as a useful adjunct to their clinical practice. The main concern related to use of the application in isolation as opposed to in conjunction with other clinical parameters. However, most clinicians felt that the application could positively reinforce or validate their clinical judgement. Conclusion Translating AI technologies into the clinical setting remains an important but challenging task. We demonstrate the effectiveness, efficiency, and system usability of a web application designed to predict COVID-19 patient outcomes from an ANN.
Editors: Eysenbach, G
Issue Date: 28-Jul-2021
Date of Acceptance: 31-May-2021
DOI: 10.2196/27992
ISSN: 2561-326X
Publisher: JMIR Publications
Start Page: 1
End Page: 13
Journal / Book Title: JMIR Formative Research
Volume: 5
Issue: 7
Copyright Statement: ©Ahmed Abdulaal, Aatish Patel, Ahmed Al-Hindawi, Esmita Charani, Saleh A Alqahtani, Gary W Davies, Nabeela Mughal, Luke Stephen Prockter Moore. Originally published in JMIR Formative Research (, 28.07.2021. This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included
Keywords: COVID-19
artificial intelligence
graphical user interface
machine learning
neural network
Publication Status: Published
Article Number: JFRms#27992
Online Publication Date: 2021-07-28
Appears in Collections:Department of Infectious Diseases
Faculty of Medicine
Imperial College London COVID-19

This item is licensed under a Creative Commons License Creative Commons