How Walmart Leverages CUPED and Reduces Experimentation Lifecycle

Moitreyee Dasgupta, Leihua Ye11 Dec 2023

Problem

Walmart's experimentation platform runs hundreds of concurrent A/B tests across web, Android and iOS. Target metrics vary a lot between customers, while the lifts product teams look for are small. Detecting such a lift takes millions of users. Running tests longer helps little: a 10 times smaller lift needs 100 times more users, and the customer base is finite. The team needed a cheaper way to reach power and end tests sooner.

Example from the article: a key business metric (not disclosed) gave different variance reduction depending on the pre-period length. A 2-week default would have cut variance by less than 1%.

Solution in brief

Walmart added CUPED to its daily analytics pipeline. The covariate is the same metric over the 28 days before the experiment starts. One coefficient θ is computed per experiment. The "de-meaned" form of the adjustment keeps group means unchanged, so the readout UI does not change. Variance fell by 5% to 20% depending on tenant, with no increase in pipeline SLA.

Questions this case answers
  • How to detect small metric lifts in online experiments when metric variance is large and extending the test is too costly
  • How to add pre-experiment variance reduction to a shared experimentation platform without breaking its daily SLA
  • How to cut the variance of A/B test metrics without running tests longer?
  • How to choose the pre-experiment window for CUPED?

Tools and methods

  • CUPED — variance reduction with a pre-experiment covariate
  • De-meaned CUPED — adjustment that keeps the original group mean
  • Apache Spark — Scala/SQL implementation of the theta and adjusted-metric steps
Continue reading — it's free

Google or a one-time email code. No password.