Understanding the collective impact of experiments

Stephane Shao, Clare Burke7 Dec 2022

Problem

Etsy teams run their own A/B tests, often on the same pages and traffic at once. Each test measures only the marginal effect of one change. Treatments can interact: two changes that each help alone can hurt together, and search, ads and recommendations take traffic from each other. The company still needs the total effect of all launches for strategic discussions and financial planning, and the sum of isolated results is not that number.

Example from the article: two made-up treatments, an orange navigation bar and an orange cart icon, each look sensible alone. Together the cart icon blends into the orange bar and becomes harder to find, which could make users abandon the purchase.

Solution in brief

Etsy runs "persistent experiments" every quarter. A small share of traffic is kept out of all experiments as a holdout. At the end of the quarter, half of the holdout gets all adopted treatments for several weeks, and an A/B test compares it with the other half on the old site. This measures the total incremental effect directly.

Questions this case answers
  • How to estimate the total effect of many concurrently launched changes when each was tested separately and their effects may interact
  • How to keep a persistent unexposed control group over a long period without freezing product development
  • How to measure the combined business impact of all changes launched in a quarter when many teams run overlapping A/B tests?
  • How to check whether the sum of individual experiment effects matches the real total effect?

Tools and methods

  • Holdout group — traffic excluded from every experiment for the whole quarter, so it keeps the old product state.
  • Comparison period — an end-of-quarter A/B test of the combined adopted treatments against the old state.
  • Persistent experiment — Etsy's name for the whole quarterly holdout scheme.
Continue reading — it's free

Google or a one-time email code. No password.