Changing predictor measurement procedures affected the performance of prediction models in clinical examples
File(s)ChangingPredictorMeasurementProcedures.pdf (720.89 KB)
Published version
Author(s)
Type
Journal Article
Abstract
OBJECTIVE: To quantify the impact of predictor measurement heterogeneity on prediction model performance. Predictor measurement heterogeneity refers to variation in the measurement of predictor(s) between the derivation of a prediction model and its validation or application. It arises, for instance, when predictors are measured using different measurement instruments or protocols. STUDY DESIGN AND SETTING: We examined effects of various scenarios of predictor measurement heterogeneity in real-world clinical examples using previously developed prediction models for diagnosis of ovarian cancer, mutation carriers for Lynch syndrome, and intrauterine pregnancy. RESULTS: Changing the measurement procedure of a predictor influenced the performance at validation of the prediction models in nine clinical examples. Notably, it induced model miscalibration. The calibration intercept at validation ranged from -0.70 to 1.43 (0 for good calibration), while the calibration slope ranged from 0.50 to 1.67 (1 for good calibration). The difference in c-statistic and scaled Brier score between derivation and validation ranged from -0.08 to +0.08 and from -0.40 to +0.16, respectively. CONCLUSION: This study illustrates that predictor measurement heterogeneity can influence the performance of a prediction model substantially, underlining that predictor measurements used in research settings should resemble clinical practice. Specification of measurement heterogeneity can help researchers explaining discrepancies in predictive performance between derivation and validation setting.
Date Issued
2020-03
Date Acceptance
2019-11-04
Citation
Journal of Clinical Epidemiology, 2020, 119, pp.7-18
ISSN
0895-4356
Publisher
Elsevier
Start Page
7
End Page
18
Journal / Book Title
Journal of Clinical Epidemiology
Volume
119
Copyright Statement
© 2019 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
Identifier
https://www.ncbi.nlm.nih.gov/pubmed/31706963
PII: S0895-4356(19)30747-4
Subjects
Prediction model
external validation
measurement error
measurement heterogeneity
predictive performance
Publication Status
Published online
Coverage Spatial
United States
Date Publish Online
2019-11-09