Problem
DoorDash Ads sells sponsored placements to restaurants and consumer packaged goods (CPG) brands, and the platform, advertisers and consumers all influence one another. A plain A/B test of a new ranking model, creative or bidding strategy breaks the stable unit treatment value assumption (SUTVA). The treatment arm can look better only because it spends more of the shared campaign budget, and a new ranking algorithm also changes auction costs for other advertisers. Shipping on such a readout risks off-target ads for consumers and fee spikes for restaurants.
Example from the article: a new automated bidding strategy tested against the old one on the same campaign budget can look better simply because it spent more.
Solution in brief
Split each campaign's budget between the experiment arms and randomize consumers per arm. Each arm then runs as its own small marketplace with a dedicated budget and cannot take budget from the other arm. This removes cannibalization and cross-arm feedback loops while keeping consumer-level randomization, which gives a much smaller minimum detectable effect than switchback tests.
Questions this case answers
- How to run a randomized experiment on a feature that allocates a shared limited budget, when treatment and control compete for the same budget and standard randomization is biased
- How to choose an experiment design for a multi-sided marketplace where interference rules out plain user-level A/B tests and time-based or cluster designs lose power
- How to A/B test a new ad bidding, pacing or ranking feature when the treatment group can eat the control group's budget?
- How to avoid biased test results in an ads marketplace where advertisers, consumers and the platform affect each other?
Tools and methods
- Budget A/B (budget split) experiment — divides a campaign's budget into per-arm pools and randomizes consumers across arms
- Experimentation platform (EP) — assigns campaigns to budget experiments and consumers to arms
- Switchback, submarket and cluster designs — alternatives DoorDash considered and rejected
- Minimum detectable effect (MDE) — the power measure used to compare designs
Google or a one-time email code. No password.