Problem
DoorDash's Dasher (courier) team tests changes in a three-sided marketplace, and three goals conflict. Orders in one area share a courier fleet, so an order-level A/B test has interference. Couriers adapt over several weeks, so retention needs a long, consistent treatment. The test also needs enough power. Each randomization unit satisfies at most two goals, so a wrong choice gives a biased or inconclusive result.
Example from the article: one change meant to improve both delivery quality and Dasher retention, where a market A/B lacks power.
Solution in brief
Pick the unit per intervention from the expected network and learning effects and a power analysis. One effect: switchback or Dasher A/B. Both effects: stratified market A/B with diff-in-diff if power allows, otherwise a switchback plus a Dasher A/B, split by markets or run in sequence.
Questions this case answers
- How to choose a randomization unit when units interfere, behavior adapts slowly and the number of clusters limits power
- How to measure a short-term and a long-term effect of the same change when one experiment design cannot capture both
- How to choose the randomization unit in a marketplace experiment when network effects, learning effects and power pull in different directions?
- How to measure both a quick delivery-quality effect and a slow supplier-retention effect of one change?
Tools and methods
- Switchback test — region and time-window randomization against interference
- Dasher A/B — courier-level test for learning effects
- Market A/B with difference-in-differences — market-level test for both effects, low power
- Stratified randomization — balances treatment and control markets
- Cluster-robust standard errors (CRSE) — valid inference with correlated deliveries
- Synthetic control — fallback when parallel trends fail
Google or a one-time email code. No password.