Problem
An ad platform has to show an advertiser how many sales its ads caused, not how many followed an ad view. Attribution also counts purchases that would have happened anyway. Randomized experiments fix this, but they are hard where an auction picks the ads. Users who would never see the advertiser's ads dilute the comparison, and removing ads for a holdout group leaves empty slots and breaks pricing.
Example from the article: a shopper looking for lettuce sees a Sponsored Products ad for salad tomatoes and buys them. Only a comparison with a similar shopper who did not see the ad shows whether that sale was incremental.
Solution in brief
Split users per advertiser into an ads group and a holdout group, and compare total sales (organic plus attributed) of the advertised products. Assign users before the auction's selection step, and use a simulated ("ghost ad") auction to compare only users who would have been exposed. Average advertiser results with spend weights for a platform metric, and run product A/B tests on top of the holdouts.
Questions this case answers
- How to estimate the causal effect of paid placements on sales when exposure is decided by an auction and many assigned users are never exposed
- How to aggregate many per-customer experiment effects into one platform-level metric that tracks changes over time
- How to measure how many sales an ad campaign actually caused, rather than sales it only got credit for?
- How to run a per-advertiser holdout experiment when ads are chosen by an auction?
Tools and methods
- Advertiser-level holdout — random per-advertiser split of users
- Roulette — Instacart's internal assignment engine
- Ghost Ad simulation — simulated auction that finds which holdout impressions would have won
- Spend-weighted aggregation — platform-level incrementality from advertiser-level results
- A/B test orthogonal to holdouts — compares system-level incrementality across product variants
Google or a one-time email code. No password.