Problem
An A/B test compares two different sets of users. Their outcomes differ not only because of the treatment but also because of who landed in each group. The difference-in-means estimate of the treatment effect is therefore unbiased but noisy. On an experimentation platform this noise shows up as wide confidence intervals and low power: tests have to run longer, or they end without a clear answer. Adding traffic helps, but it is limited and every extra week delays the decision.
Example from the article: in ExP's repeated simulation with a known true effect of 2.5, some random splits put users with higher pre-experiment values into one group. On those splits difference-in-means lands far from 2.5, while the CUPED estimate on the same data stays closer to it.
Solution in brief
Variance reduction (VR) changes the estimator, not the data: the metric is adjusted by the prediction of a linear regression on pre-experiment covariates. The estimator's variance is multiplied by (at , 60% of the original), and the estimate stays consistent.
Questions this case answers
- How to reduce the variance of a treatment effect estimate in a randomized comparison using pre-experiment covariates so that decisions come faster
- How to explain the gain from a variance-reduction estimator to experiment owners as equivalent extra traffic
- How to reduce the variance of the treatment effect estimate in an A/B test to reach decisions faster
- How to explain the gain from CUPED to experiment owners in terms of extra traffic
Tools and methods
- CUPED (Controlled experiment Using Pre-Experiment Data) — regression adjustment of the metric on pre-experiment data.
- ANCOVA2 (analysis of covariance) / Lin's estimator — a regression with a covariate-by-group interaction, equivalent to CUPED.
- Effective traffic multiplier — , computed for each metric.
Google or a one-time email code. No password.