Unlocking 66% Faster Experiments with Covariate Adjustment: Power Analysis for Clustered Data, Ratio Metrics, and OLS Regression

Benjamin Knight, Tim Hesterberg17 Jun 2024

Problem

Experiments at Instacart are randomized by customer or shopper, but many metrics are measured per order or are ratios, such as revenue share of one category. Orders of one customer are correlated, so ignoring clustering gives standard errors that are too small. Off-the-shelf power calculators handle only independent observations and plain means, and they ignore covariate adjustment. Tests planned with them run longer than needed, and tests on ratio metrics can be badly under-powered.

Example from the article: a test of store planogram data for shoppers, randomized by shopper, with 10 million orders and a target lift of $0.05 GTV per order. A plain t-test power calculation gives 15% power.

Solution in brief

Every case reduces to one recipe: compute a per-unit residual, take its sample standard deviation sr, and plug sr (divided by the mean denominator for ratios) into the usual power formula in place of sy. For ratios the residual comes from the delta method, and for covariate adjustment it comes from a regression. The same recipe covers both together.

Questions this case answers
  • How to plan the sample size of a randomized experiment when the metric is a ratio of two random quantities and observations are clustered within the randomization unit
  • How to account for pre-experiment covariates in power analysis when standard power calculators only support independent observations and simple means
  • How to calculate sample size and MDE for an A/B test on a ratio metric or a metric measured per order while randomizing by customer?
  • How to include covariate adjustment (CUPED) in power analysis so that planned run time reflects the variance reduction?

Tools and methods

  • Power analysis / MDE — relate sample size, alpha, power and minimum detectable effect.
  • Delta method on cluster-level ratios — correct standard errors for ratio metrics and clustered orders.
  • Regression (covariate) adjustment — CUPED-style variance reduction with pre-experiment covariates.
Continue reading — it's free

Google or a one-time email code. No password.