Problem
Walmart's experimentation platform runs hundreds of concurrent A/B tests across web, Android and iOS. Target metrics vary a lot between customers, while the lifts product teams look for are small. Detecting such a lift takes millions of users. Running tests longer helps little: a 10 times smaller lift needs 100 times more users, and the customer base is finite. The team needed a cheaper way to reach power and end tests sooner.
Example from the article: a key business metric (not disclosed) gave different variance reduction depending on the pre-period length. A 2-week default would have cut variance by less than 1%.
Solution in brief
Walmart added CUPED to its daily analytics pipeline. The covariate is the same metric over the 28 days before the experiment starts. One coefficient is computed per experiment. The "de-meaned" form of the adjustment keeps group means unchanged, so the readout UI does not change. Variance fell by 5% to 20% depending on tenant, with no increase in pipeline SLA.
Questions this case answers
- How to detect small metric lifts in online experiments when metric variance is large and extending the test is too costly
- How to add pre-experiment variance reduction to a shared experimentation platform without breaking its daily SLA
- How to cut the variance of A/B test metrics without running tests longer?
- How to choose the pre-experiment window for CUPED?
Tools and methods
- CUPED — variance reduction with a pre-experiment covariate
- De-meaned CUPED — adjustment that keeps the original group mean
- Apache Spark — Scala/SQL implementation of the theta and adjusted-metric steps
Google or a one-time email code. No password.