Problem
Advertisers ask for segments such as brand-loyal customers or people likely to buy a product type soon. Positives are a tiny share of millions of customers, so the team down-samples the classes before training. That shifts the class priors, and the scores stop reflecting true likelihoods: a score of 0.99 does not mean 99% of those customers will convert. Bids, error estimates and model combinations all stop working. Multiplying two uncorrected scores can give a wrong ranking and a wrong segment.
Example from the article: two Walmart Adtech models, one predicting loyalty to a brand and one predicting purchase of a product type soon, were multiplied to rank customers. Calibrating both first gave a segment with 17% higher revenue.
Solution in brief
Check calibration on a holdout set with a reliability curve, then correct the scores with the Pozzolo et al. (2015) prior-shift formula, which rescales each class by the inverse of its down-sampling ratio.
Questions this case answers
- How to recover true class probabilities from a classifier trained on a down-sampled, class-balanced sample so that scores from several models can be multiplied and ranked
- How to check whether predicted scores match observed outcome rates when only a holdout set with the original class balance is available
- How to turn scores from a model trained on down-sampled data into real probabilities?
- How to combine scores from two models into one ranking without distorting it?
Tools and methods
- Reliability curve — bins of mean predicted score against fraction of positives, to see miscalibration
- Pozzolo et al. prior-shift correction — converts scores from a down-sampled model to original-prior probabilities
- Logistic regression — the probabilistic model whose scores were calibrated
- Platt scaling, isotonic regression — mentioned for models that stay miscalibrated even on balanced data
Google or a one-time email code. No password.