Problem
In a delivery marketplace the dispatch system matches orders to shoppers globally, so treating one order changes outcomes for its neighbors. Randomizing single orders contaminates the control group, so teams randomize whole regions, often with a switchback design that re-randomizes each region daily. The number of independent units stays small and power is low. At Instacart a powered experiment can take months, and for very noisy metrics it may be infeasible. Standard CUPED on region-day features cuts variance by only up to 10%.
Example from the article: the metric is late-delivery percentage per region-day, and most of its variation comes from order characteristics, not region-level features.
Solution in brief
Keep the coarse randomization, but build the CUPED covariate at the fine grain. Train an order-level model on pre-treatment features, aggregate its predictions to the region-day the same way the metric is aggregated, and use that aggregate as the CUPED covariate. A placebo test on the aggregated prediction checks that no feature is affected by the treatment.
Questions this case answers
- How to reduce the variance of an experiment metric when interference forces randomization at a coarse cluster level but outcomes are observed at a much finer unit
- How to shorten marketplace experiments that must be randomized by region because orders interfere with each other?
- How to get more out of CUPED when the covariate has to be built at the coarse randomization level?
- How to use an order-level ML model for variance reduction without biasing the treatment effect?
Tools and methods
- CUPED / CUPAC — variance reduction with a pre-treatment covariate; here the covariate is an ML prediction
- Switchback test — region-day randomization that the adjustment sits on top of
- Gradient boosted tree — order-level model predicting whether an order will be late
- Placebo test — mean difference test on the predicted outcome, which must show no effect
Google or a one-time email code. No password.