Journal
STATISTICAL METHODS IN MEDICAL RESEARCH
Volume 27, Issue 11, Pages 3505-3522Publisher
SAGE PUBLICATIONS LTD
DOI: 10.1177/0962280217705678
Keywords
Validation; performance statistics; C-statistic; discrimination; calibration; meta-analysis; between-study distribution; heterogeneity; simulation
Categories
Funding
- MRC Midlands Hub for Trials Methodology Research (Medical Research Council) [G0800808]
- MRC Methodology Research Programme [MR/J013595/1]
- Netherlands Organization for Scientific Research [91617050]
- MRC [G0800808, MR/J013595/1, MR/J013595/2] Funding Source: UKRI
- Medical Research Council [G0800808, MR/J013595/2, 1587806, MR/J013595/1] Funding Source: researchfish
Ask authors/readers for more resources
If individual participant data are available from multiple studies or clusters, then a prediction model can be externally validated multiple times. This allows the model's discrimination and calibration performance to be examined across different settings. Random-effects meta-analysis can then be used to quantify overall (average) performance and heterogeneity in performance. This typically assumes a normal distribution of true' performance across studies. We conducted a simulation study to examine this normality assumption for various performance measures relating to a logistic regression prediction model. We simulated data across multiple studies with varying degrees of variability in baseline risk or predictor effects and then evaluated the shape of the between-study distribution in the C-statistic, calibration slope, calibration-in-the-large, and E/O statistic, and possible transformations thereof. We found that a normal between-study distribution was usually reasonable for the calibration slope and calibration-in-the-large; however, the distributions of the C-statistic and E/O were often skewed across studies, particularly in settings with large variability in the predictor effects. Normality was vastly improved when using the logit transformation for the C-statistic and the log transformation for E/O, and therefore we recommend these scales to be used for meta-analysis. An illustrated example is given using a random-effects meta-analysis of the performance of QRISK2 across 25 general practices.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available