Balancing Network Effects, Learning Effects, and Power in Experiments

Wei Feng, Jared Bauman16 Feb 2022

Problem

DoorDash's Dasher (courier) team tests changes in a three-sided marketplace, and three goals conflict. Orders in one area share a courier fleet, so an order-level A/B test has interference. Couriers adapt over several weeks, so retention needs a long, consistent treatment. The test also needs enough power. Each randomization unit satisfies at most two goals, so a wrong choice gives a biased or inconclusive result.

Example from the article: one change meant to improve both delivery quality and Dasher retention, where a market A/B lacks power.

Solution in brief

Pick the unit per intervention from the expected network and learning effects and a power analysis. One effect: switchback or Dasher A/B. Both effects: stratified market A/B with diff-in-diff if power allows, otherwise a switchback plus a Dasher A/B, split by markets or run in sequence.

Questions this case answers
  • How to choose a randomization unit when units interfere, behavior adapts slowly and the number of clusters limits power
  • How to measure a short-term and a long-term effect of the same change when one experiment design cannot capture both
  • How to choose the randomization unit in a marketplace experiment when network effects, learning effects and power pull in different directions?
  • How to measure both a quick delivery-quality effect and a slow supplier-retention effect of one change?

Tools and methods

  • Switchback test — region and time-window randomization against interference
  • Dasher A/B — courier-level test for learning effects
  • Market A/B with difference-in-differences — market-level test for both effects, low power
  • Stratified randomization — balances treatment and control markets
  • Cluster-robust standard errors (CRSE) — valid inference with correlated deliveries
  • Synthetic control — fallback when parallel trends fail
Continue reading — it's free

Google or a one-time email code. No password.