Problem
Snap picks one ad per impression from millions in an auction that uses model scores, bids and budgets. Advertisers are billed per impression but bid for other events such as swipe-up or app install, so predicted probabilities must be accurate in absolute terms, not only in rank. Three things get in the way: models learn only from ads that already won an auction, conversions arrive up to a few weeks after the impression, and the ad inventory turns over fast.
Example from the article: an app-purchase can happen weeks after the ad is shown and can repeat, so a label that looks negative today may turn positive later.
Solution in brief
Snap scores each candidate with a multi-task deep model and corrects calibration with a separate layer on top. The model is warm-started and updated hourly to daily. Changes are tested in a budget-split A/B test: each advertiser's budget is cut into N parts and a change touches only one part.
Questions this case answers
- How to keep predicted probabilities calibrated when selection by the highest score inflates them on live traffic
- How to update a prediction model on outcomes that are reported with a delay of days to weeks without learning from false negatives
- How to measure the effect of a model change when the test and control arms draw on one shared budget
- How to keep ad conversion predictions calibrated when the auction picks only the highest scores?
Tools and methods
- Multi-task models (MMoE, PLE) — predict install, purchase and sign-up jointly
- Feature-interaction layers (DCN, DCN v2) — feature crosses
- Calibration correction layer — Platt scaling, isotonic regression or a small neural network on model scores
- Budget-split A/B test — unbiased measurement in a two-sided marketplace
- Continuous warm-start updates — adapt to new ad ids and fresh labels
- TensorFlow, Kubeflow pipelines — training and orchestration
Google or a one-time email code. No password.