Problem
Experiments at Instacart are randomized by customer or shopper, but many metrics are measured per order or are ratios, such as revenue share of one category. Orders of one customer are correlated, so ignoring clustering gives standard errors that are too small. Off-the-shelf power calculators handle only independent observations and plain means, and they ignore covariate adjustment. Tests planned with them run longer than needed, and tests on ratio metrics can be badly under-powered.
Example from the article: a test of store planogram data for shoppers, randomized by shopper, with 10 million orders and a target lift of $0.05 GTV per order. A plain t-test power calculation gives 15% power.
Solution in brief
Every case reduces to one recipe: compute a per-unit residual, take its sample standard deviation , and plug (divided by the mean denominator for ratios) into the usual power formula in place of . For ratios the residual comes from the delta method, and for covariate adjustment it comes from a regression. The same recipe covers both together.
Questions this case answers
- How to plan the sample size of a randomized experiment when the metric is a ratio of two random quantities and observations are clustered within the randomization unit
- How to account for pre-experiment covariates in power analysis when standard power calculators only support independent observations and simple means
- How to calculate sample size and MDE for an A/B test on a ratio metric or a metric measured per order while randomizing by customer?
- How to include covariate adjustment (CUPED) in power analysis so that planned run time reflects the variance reduction?
Tools and methods
- Power analysis / MDE — relate sample size, alpha, power and minimum detectable effect.
- Delta method on cluster-level ratios — correct standard errors for ratio metrics and clustered orders.
- Regression (covariate) adjustment — CUPED-style variance reduction with pre-experiment covariates.
Google or a one-time email code. No password.