Development, Calibration Repair and Cross-Study Transportability of a Genomics-Informed 12-Month Mortality Model for Glioma

Abstract

Jacopo Martelli*

Background: Early mortality in glioma is driven by age and by molecular markers, notably IDH1 mutation, but most published prognostic tools rely on curated trial or single-institution data. Public genomic cohorts could support broader models, yet their heterogeneity makes pooled accuracy hard to interpret, and the calibration of such models is rarely audited.

Methods: We developed gradient-boosting and logistic-regression models for 12-month all-cause mortality (a landmark label) in 3,590 patients with glioma from 24 public Bio Portal studies, using age, sex, a 45-gene mutation panel, panel mutation count and a sequencing indicator. Performance was assessed by 5-fold out-of-fold cross-validation and by leave-one-study-out validation in the six largest studies. To audit the pipeline, we reconstructed its out-of-fold score distribution from the exported ROC steps, which reproduced the reported Brier score, calibration slope and AUC. We then compared calibration against a null model, repaired it by a parameter-free prior-shift correction and by cross-validated Platt and isotonic recalibration, derived operating characteristics and decision-curve net benefit, meta- analysed held-out-study AUCs, regressed them on study mortality, and decomposed pooled discrimination into within- study and between-study case–control pairs.

Results: 770 patients (21.4%) died within 12 months. The gradient-boosting model reached AUC 0.774 (95% CI 0.756– 0.792), against 0.734 for age and sex alone, and was not distinguishable from logistic regression (0.770). Its raw probabilities were badly mis calibrated (mean predicted risk 0.40 against observed 0.21, calibration slope 0.71): The Brier score of 0.190 was worse than that of a model predicting the prevalence for everyone (0.168, skill −13.0%). A prior-shift correction implied by class weighting, with no fitted parameters, reduced the Brier score to 0.143 (skill +15.4%) and the expected calibration error from 0.187 to 0.026, cross-validated Platt scaling gave 0.142 and 0.018. After recalibration the model exceeded the best default strategy at threshold probabilities from 0.08 to 0.48 (95% bootstrap band) and avoided 32 unnecessary actions per 100 patients at a threshold of 0.20. Discrimination in held- out studies was heterogeneous (AUC 0.685–0.843, random-effects pooled 0.761, 95% CI 0.705–0.809, I2 69%, 95% prediction interval 0.578–0.881) and fell steeply with study mortality (Spearman ρ = −0.94). Only 14.5% of all case– control pairs lay within a single study, within-study discrimination was about 0.73, against 0.78 for between-study pairs. Conclusions: A compact mutation panel with age and sex carries real prognostic information about 12-month glioma mortality, age and IDH1 account for 74% of the importance among the top 15 features.

The headline pooled AUC nevertheless overstates what a new site should expect, the raw probabilities must not be used, and validation by institution is required before any clinical use. The secondary analyses rest on the pipeline's exported summaries, not on patient-level predictions.

HTML PDF