Mitigating the winner's curse in online experiments

Stephane Shao13 Dec 2022

Problem

Product teams ship a change when its A/B test shows a statistically significant positive lift, then report that observed lift as the change's business value. The detection rule is sound, but the size estimate is biased: among experiments that pass the threshold, the observed lift overestimates the true lift on average. Finance and planning then count on gains that never fully show up. Etsy calls this the winner's curse. It is a selection effect of the decision protocol, not of analyst behaviour such as p-hacking.

Example from the article: two Etsy experiments both show an observed lift of 1.6%, but one has 1.8% standard error and the other 0.2%, so they deserve very different trust.

Solution in brief

Treat the observed lift as one noisy signal and combine it with a prior built from the lifts of past experiments, using a Bayesian model. The posterior mean is a "discounted lift": a weighted average of the observed lift and the historical expectation. Noisy experiments are pulled strongly toward the prior; precise ones barely change.

Questions this case answers
  • How to correct the upward bias in effect estimates that appears when only statistically significant results are selected and reported
  • How to combine the historical distribution of effect sizes with a noisy new estimate to get a calibrated effect size
  • How to report the true business impact of A/B test wins without overstating it?
  • How to discount observed lifts of statistically significant experiments?

Tools and methods

  • A/B test with significance threshold — the selection rule that creates the bias
  • Prior fitted on historical lifts — mixture of light- and heavy-tailed distributions (after Deng et al.)
  • Gibbs sampling (MCMC) — draws the full posterior of the true lift
  • Discounted lift — posterior mean, reported instead of the raw lift
Continue reading — it's free

Google or a one-time email code. No password.