Deep Dive Into Variance Reduction

Laura Cosgrove, Jen Townsend, Jonathan Litz15 Nov 2022

Problem

An A/B test compares two different sets of users. Their outcomes differ not only because of the treatment but also because of who landed in each group. The difference-in-means estimate of the treatment effect is therefore unbiased but noisy. On an experimentation platform this noise shows up as wide confidence intervals and low power: tests have to run longer, or they end without a clear answer. Adding traffic helps, but it is limited and every extra week delays the decision.

Example from the article: in ExP's repeated simulation with a known true effect of 2.5, some random splits put users with higher pre-experiment values into one group. On those splits difference-in-means lands far from 2.5, while the CUPED estimate on the same data stays closer to it.

Solution in brief

Variance reduction (VR) changes the estimator, not the data: the metric is adjusted by the prediction of a linear regression on pre-experiment covariates. The estimator's variance is multiplied by (1−R2) (at R2=0.4, 60% of the original), and the estimate stays consistent.

Questions this case answers
  • How to reduce the variance of a treatment effect estimate in a randomized comparison using pre-experiment covariates so that decisions come faster
  • How to explain the gain from a variance-reduction estimator to experiment owners as equivalent extra traffic
  • How to reduce the variance of the treatment effect estimate in an A/B test to reach decisions faster
  • How to explain the gain from CUPED to experiment owners in terms of extra traffic

Tools and methods

  • CUPED (Controlled experiment Using Pre-Experiment Data) — regression adjustment of the metric on pre-experiment data.
  • ANCOVA2 (analysis of covariance) / Lin's estimator — a regression with a covariate-by-group interaction, equivalent to CUPED.
  • Effective traffic multiplier — 1/(1−R2), computed for each metric.
Continue reading — it's free

Google or a one-time email code. No password.