Calibration of the Prediction Scores: How does it help?

Arpan Maheshwari20 May 2022

Problem

Advertisers ask for segments such as brand-loyal customers or people likely to buy a product type soon. Positives are a tiny share of millions of customers, so the team down-samples the classes before training. That shifts the class priors, and the scores stop reflecting true likelihoods: a score of 0.99 does not mean 99% of those customers will convert. Bids, error estimates and model combinations all stop working. Multiplying two uncorrected scores can give a wrong ranking and a wrong segment.

Example from the article: two Walmart Adtech models, one predicting loyalty to a brand and one predicting purchase of a product type soon, were multiplied to rank customers. Calibrating both first gave a segment with 17% higher revenue.

Solution in brief

Check calibration on a holdout set with a reliability curve, then correct the scores with the Pozzolo et al. (2015) prior-shift formula, which rescales each class by the inverse of its down-sampling ratio.

Questions this case answers
  • How to recover true class probabilities from a classifier trained on a down-sampled, class-balanced sample so that scores from several models can be multiplied and ranked
  • How to check whether predicted scores match observed outcome rates when only a holdout set with the original class balance is available
  • How to turn scores from a model trained on down-sampled data into real probabilities?
  • How to combine scores from two models into one ranking without distorting it?

Tools and methods

  • Reliability curve — bins of mean predicted score against fraction of positives, to see miscalibration
  • Pozzolo et al. prior-shift correction — converts scores from a down-sampled model to original-prior probabilities
  • Logistic regression — the probabilistic model whose scores were calibrated
  • Platt scaling, isotonic regression — mentioned for models that stay miscalibrated even on balanced data
Continue reading — it's free

Google or a one-time email code. No password.