GenAI exceeds clinical experts in predicting acute kidney injury following paediatric cardiopulmonary bypass
File(s)
Published version
Author(s)
Type
Journal Article
Abstract
The emergence of large language models (LLMs) opens new horizons to leverage, often unused, information in clinical text. Our study aims to capitalise
on this new potential. Specifically, we examine the utility of text embeddings
generated by LLMs in predicting postoperative acute kidney injury (AKI)
in paediatric cardiopulmonary bypass (CPB) patients using electronic health
record (EHR) text, and propose methods for explaining their output. AKI
could be a serious complication in paediatric CPB and its accurate prediction can significantly improve patient outcomes by enabling timely interventions. We evaluate various text embedding algorithms such as Doc2Vec, top performing sentence transformers on Hugging Face, and commercial LLMs
from Google and OpenAI. We benchmark the cross-validated performance
of these ‘AI models’ against a ‘baseline model’ as well as an established
clinically-defined ‘expert model’. The baseline model includes structured
features, i.e., patient gender, age, height, body mass index and length of
operation. The majority of AI models surpass, not only the baseline model,
but also the expert model. An ensemble of AI and clinical-expert models improves discriminative performance by 23% compared to the baseline model.
Consistency of patient clusters formed from AI-generated embeddings with
clinical-expert clusters - measured via the adjusted rand index and adjusted
mutual information metrics - illustrates the medical validity of LLM embeddings. We create a reverse mapping from the numeric embedding space to the natural-language domain via the embedding-based clusters, generating
medical labels for the clusters in the process. We also use text-generating
LLMs to summarise the differences between AI and expert clusters. Such
‘explainability’ outputs can increase medical practitioners’ trust in the AI
applications, and help generate new hypotheses, e.g., by studying the association of cluster memberships and outcomes of interest.
on this new potential. Specifically, we examine the utility of text embeddings
generated by LLMs in predicting postoperative acute kidney injury (AKI)
in paediatric cardiopulmonary bypass (CPB) patients using electronic health
record (EHR) text, and propose methods for explaining their output. AKI
could be a serious complication in paediatric CPB and its accurate prediction can significantly improve patient outcomes by enabling timely interventions. We evaluate various text embedding algorithms such as Doc2Vec, top performing sentence transformers on Hugging Face, and commercial LLMs
from Google and OpenAI. We benchmark the cross-validated performance
of these ‘AI models’ against a ‘baseline model’ as well as an established
clinically-defined ‘expert model’. The baseline model includes structured
features, i.e., patient gender, age, height, body mass index and length of
operation. The majority of AI models surpass, not only the baseline model,
but also the expert model. An ensemble of AI and clinical-expert models improves discriminative performance by 23% compared to the baseline model.
Consistency of patient clusters formed from AI-generated embeddings with
clinical-expert clusters - measured via the adjusted rand index and adjusted
mutual information metrics - illustrates the medical validity of LLM embeddings. We create a reverse mapping from the numeric embedding space to the natural-language domain via the embedding-based clusters, generating
medical labels for the clusters in the process. We also use text-generating
LLMs to summarise the differences between AI and expert clusters. Such
‘explainability’ outputs can increase medical practitioners’ trust in the AI
applications, and help generate new hypotheses, e.g., by studying the association of cluster memberships and outcomes of interest.
Date Issued
2025-07-01
Date Acceptance
2025-05-28
Citation
Scientific Reports, 2025, 15
ISSN
2045-2322
Publisher
Nature Portfolio
Journal / Book Title
Scientific Reports
Volume
15
Copyright Statement
© The Author(s) 2025 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
License URL
Identifier
10.1038/s41598-025-04651-8
Subjects
Generative artificial intelligence
Text embedding
Electronic health records
Cardiopulmonary bypass
Acute kidney injury
Spherical k-means
Publication Status
Published
Article Number
20847
Date Publish Online
2025-07-01