Validation and updating of risk models based on multinomial logistic regression
File(s)Van-calster-2017-Validation-and-updating-of-risk-mod.pdf (701.02 KB)
Published version
Author(s)
Type
Journal Article
Abstract
Background: Risk models often perform poorly at external validation in terms of discrimination or calibration. Updating methods are needed to improve performance of multinomial logistic regression models for risk prediction. Methods: We consider simple and more refined updating approaches to extend previously proposed methods for dichotomous outcomes. These include model recalibration (adjustment of intercept and/or slope), revision (re-estimation of individual model coefficients), and extension (revision with additional markers). We suggest a closed testing procedure to assist in deciding on the updating complexity. These methods are demonstrated on a case study of women with pregnancies of unknown location (PUL). A previously developed risk model predicts the probability that a PUL is a failed, intra-uterine, or ectopic pregnancy. We validated and updated this model on more recent patients from the development setting (temporal updating; n = 1422) and on patients from a different hospital (geographical updating; n = 873). Internal validation of updated models was performed through bootstrap resampling. Results: Contrary to dichotomous models, we noted that recalibration can also affect discrimination for multinomial risk models. If the number of outcome categories is higher than the number of variables, logistic recalibration is obsolete because straightforward model refitting does not require the estimation of more parameters. Although recalibration strongly improved performance in the case study, the closed testing procedure selected model revision. Further, revision of functional form of continuous predictors had a positive effect on discrimination, whereas penalized estimation of changes in model coefficients was beneficial for calibration. Conclusions: Methods for updating of multinomial risk models are now available to improve predictions in new settings. A closed testing procedure is helpful to decide whether revision is preferred over recalibration. Because multicategory outcomes increase the number of parameters to be estimated, we recommend full model revision only when the sample size for each outcome category is large.
Date Issued
2017-02-08
Date Acceptance
2016-09-09
Citation
Diagnostic and Prognostic Research, 2017, 1
ISSN
2397-7523
Publisher
BMC
Journal / Book Title
Diagnostic and Prognostic Research
Volume
1
Copyright Statement
© The Author(s). 2017. This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link tothe Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Sponsor
Imperial College Healthcare NHS Trust- BRC Funding
Chelsea & Westminster Hospital NHS Foundation Trust
Identifier
https://www.ncbi.nlm.nih.gov/pubmed/31093534
PII: 2
Grant Number
RDD03 79560
RDIP0033 RAMP
Subjects
Calibration
Discrimination
Model updating
Multicategory outcome
Multinomial logistic regression
Prediction models
Risk models
Publication Status
Published
Coverage Spatial
England
Article Number
ARTN 2