Problem
Instacart ranks Sponsored Product ads with a Wide & Deep model that predicts the click probability (pCTR). Ads are ranked by effective cost per thousand impressions (eCPM), which depends on the bid and on pCTR, so a miscalibrated pCTR puts ads in the wrong order and misprices clicks. Filtering thresholds and ranking weights also rely on the raw probability. Calibration data is hard to get, because logs of ranked ads carry selection and position biases created by the earlier model.
Example from the article: the model is trained on ranked impressions, but its earlier predictions decided which ads were shown and where, so the labels are biased.
Solution in brief
Train the model as usual on ranked impressions. Then freeze the embeddings and lower layers and fine-tune only the top layers, mainly the final sigmoid layer, on a small hold-back sample of traffic that was not ranked by the production model. This recalibrates the same model, with no separate calibration model to deploy.
Questions this case answers
- How to make the predicted probabilities of a deep model match observed frequencies when the training data was produced by the model's own earlier decisions
- How to recalibrate a neural classifier without adding a second model to the serving path
- How to calibrate the predicted click-through rate of a deep ads ranking model when production logs are biased?
- How to measure whether predicted click probabilities match real click frequencies?
Tools and methods
- Hold-back traffic — requests served without the production ranker, used as less biased ground truth
- Transfer learning (fine-tuning top layers) — the calibration step proposed by the authors
- Calibration score, reliability diagram, Expected Calibration Error (ECE) — metrics to measure calibration
- Platt scaling, Platt scaling with features, isotonic regression — baselines compared against
Google or a one-time email code. No password.