Problem
Avito, a classified-ads platform, started selling advertising to sellers who want more clicks. Ads were ranked by historical CTR, and the goal was to grow CTR and, if possible, revenue too. The team built a CTR model, and its offline ROC AUC rose from 0.52 to 0.59. The first A/B test was still grey: neither CTR nor revenue moved. Production features differed from training, and predicted probabilities were far too high.
Example from the article: in the test group the predicted CTR sat around 0.2 to 0.5 with a peak near 0.3, while the control group was close to zero.
Solution in brief
Calibrate the CTR model, then pick the calibration variant that sits between "rank by bid" (revenue) and "rank by model score" (clicks). With calibrated probabilities, the team added two more levers: hiding impressions below a probability threshold and lifting ads that the model expects to be clicked. Each step produced a positive A/B result.
Questions this case answers
- How to calibrate predicted click probabilities so that a ranking balances clicks against revenue when offline ranking quality improves but online results do not
- How to use calibrated click probabilities to decide whether to show an item at all and how strongly to highlight it
- How to raise ad CTR and revenue at the same time when ads are ranked by bid or historical CTR
- How to fix a CTR model that gains offline ROC AUC but shows a grey A/B test
Tools and methods
- CTR prediction model — features from user, user-item, user-banner and banner tables
- Probability calibration — sigmoid, negative exponent, logistic regression, isotonic and others, compared on a bid-versus-score plane
- Probability threshold — hides ads the model expects to get no click
- Position re-ranking — raises visible ads on the better-performing side of the feed
- Feature store, Kubeflow-based platform (Aviflow) — serving and monitoring
Google or a one-time email code. No password.