Clinician acceptance of AI-generated discharge summaries
File(s)
Author(s)
Type
Journal Article
Abstract
Importance
Hospital discharge summaries are essential for continuity of care, but fragmented electronic health record (EHR) data can compromise their quality and safety. Large language models (LLMs) are increasingly proposed for clinical documentation tasks, despite limited evidence regarding their performance in healthcare settings compared with the current standard of care.
Objective
We evaluated the clinician acceptability of GPT-4o-mini-generated discharge summaries derived from EHRs compared with routine clinician-written summaries.
Design
Retrospective non-inferiority study comparing LLM-generated with clinician-authored discharge summaries. Data analysis were conducted from July 2025 to February 2026.
Setting
Admission records from five London hospitals (Sept 1, 2022, to Sept 30, 2023).
Participants
1294 summaries (645 LLM-generated; 649 clinician-authored) were evaluated against the full admission record by 12 practising doctors.
Intervention
LLM-generated summaries were produced using GPT-4o-mini (2024-07-18) and a clinician co-designed prompt.
Main Outcome and Measures
The primary outcome was clinician sign-off with no or minor edits, derived from two sequential questions; whether the reviewer would sign the summary, and whether no, minor or major edits were required. A 5-percentage-point non-inferiority margin was prespecified. Secondary outcomes assessed accuracy, completeness, tone, clarity, and potential harmfulness of missing or incorrect information on a 5-point Likert scale.
Results
The study sample comprised 1,294 unique cases (649 clinician-authored and 645 LLM-generated); median age was 46.0 years [IQR 26.2–69.8], 56.6% were assigned female at birth, and median length of stay was 3.0 days [IQR 1.0–7.0]. The absolute difference in model-predicted probability of sign-off with no or minor edits (LLM-generated minus clinician-authored) was 22.2 percentage points (95% CI 19.0-27.5); the lower bound exceeded the prespecified non-inferiority margin of -5 percentage points, demonstrating non-inferiority. LLM-generated summaries received higher ratings for accuracy, completeness, tone, and clarity (adjusted risk differences for score ≥4 were 19.4–29.1 percentage points; all 95% CIs excluded zero) and had a lower proportion of cases rated ≥3 for potential harm (17.2-19.4%) compared with clinician-authored summaries (37.5-38.5%).
Conclusion and Relevance
In this quality improvement study, LLM-generated discharge summaries met the current standard of care, representing a promising early use case for generative artificial intelligence (AI) in healthcare with close clinician oversight.
Hospital discharge summaries are essential for continuity of care, but fragmented electronic health record (EHR) data can compromise their quality and safety. Large language models (LLMs) are increasingly proposed for clinical documentation tasks, despite limited evidence regarding their performance in healthcare settings compared with the current standard of care.
Objective
We evaluated the clinician acceptability of GPT-4o-mini-generated discharge summaries derived from EHRs compared with routine clinician-written summaries.
Design
Retrospective non-inferiority study comparing LLM-generated with clinician-authored discharge summaries. Data analysis were conducted from July 2025 to February 2026.
Setting
Admission records from five London hospitals (Sept 1, 2022, to Sept 30, 2023).
Participants
1294 summaries (645 LLM-generated; 649 clinician-authored) were evaluated against the full admission record by 12 practising doctors.
Intervention
LLM-generated summaries were produced using GPT-4o-mini (2024-07-18) and a clinician co-designed prompt.
Main Outcome and Measures
The primary outcome was clinician sign-off with no or minor edits, derived from two sequential questions; whether the reviewer would sign the summary, and whether no, minor or major edits were required. A 5-percentage-point non-inferiority margin was prespecified. Secondary outcomes assessed accuracy, completeness, tone, clarity, and potential harmfulness of missing or incorrect information on a 5-point Likert scale.
Results
The study sample comprised 1,294 unique cases (649 clinician-authored and 645 LLM-generated); median age was 46.0 years [IQR 26.2–69.8], 56.6% were assigned female at birth, and median length of stay was 3.0 days [IQR 1.0–7.0]. The absolute difference in model-predicted probability of sign-off with no or minor edits (LLM-generated minus clinician-authored) was 22.2 percentage points (95% CI 19.0-27.5); the lower bound exceeded the prespecified non-inferiority margin of -5 percentage points, demonstrating non-inferiority. LLM-generated summaries received higher ratings for accuracy, completeness, tone, and clarity (adjusted risk differences for score ≥4 were 19.4–29.1 percentage points; all 95% CIs excluded zero) and had a lower proportion of cases rated ≥3 for potential harm (17.2-19.4%) compared with clinician-authored summaries (37.5-38.5%).
Conclusion and Relevance
In this quality improvement study, LLM-generated discharge summaries met the current standard of care, representing a promising early use case for generative artificial intelligence (AI) in healthcare with close clinician oversight.
Date Acceptance
2026-09-08
Citation
JAMA Network Open
ISSN
2574-3805
Publisher
JAMA Network
Journal / Book Title
JAMA Network Open
Copyright Statement
Copyright This paper is embargoed until publication. Once published the Version of Record (VoR) will be available on immediate open access.
License URL
Publication Status
Accepted