When Metrics Lie: Countering Domain Shift in MRI

Margarita Benedichuk23 May 2026

Problem

A classifier for brain MRI passes validation and then degrades at a new clinic, because scanners and protocols differ between sites. Aggregate metrics hide this, yet in clinical use each patient matters: a wrong prediction means toxic useless chemotherapy or withheld effective therapy. The team works with doctors across Siberia, and every new clinic brings its own shift. They need to know where the model stops being applicable.

Example from the article: predicting MGMT status (a marker of tumor response to chemotherapy) from four MRI modalities. Published metrics could not be reproduced, and accuracy of the base models under domain shift was about 47%, worse than a coin flip.

Solution in brief

Run a second, interpretable branch next to the neural pipeline: extract radiomic features, which capture the scanner's "fingerprint", and use them as a domain-shift detector. Then predict, case by case, the chance that the main pipeline errs (Radiomics Reliability Score, RRS) and send high-risk cases to a doctor instead of the automatic path.

Questions this case answers
  • How to detect that a classifier is applied to data from a shifted source where aggregate validation metrics look fine
  • How to estimate for each case which prediction is likely wrong and route risky cases to a human
  • How to detect domain shift between training data and a new site when aggregate validation metrics look fine
  • How to estimate in advance which individual cases a classifier will probably get wrong

Tools and methods

  • Radiomics (PyRadiomics, 860 features) — whole-image texture and intensity features, computed in seconds.
  • PCA of radiomic features — interpretable axes of case difficulty (PC1–PC5).
  • Population Stability Index (PSI) — distribution difference between sites.
  • Model × patient table with cluster-robust logistic regression and GLMM — tests what explains errors.
  • Ensemble agreement over 64 models — finds systematically hard cases.
  • RRS meta-model — predicts error rate from PC1–PC5.
Continue reading — it's free

Google or a one-time email code. No password.