Problem
Teams planning an A/B test need a required sample size, but historical data often misdescribes the outcome distribution. At Spotify the user base is diverse and keeps changing, especially in new markets and in new-user experiments. Historical data also says nothing about the effect of a variant that has never run, and effects differ across users with different listening habits. Sequential tests fix peeking but lose power and bias the effect estimate, and an extra A/A test lengthens the experiment without guaranteeing an accurate sample size.
Illustrative example: a team tests a feature for new users in a new market. Past data comes from other users, so the planned sample size may be wrong.
Solution in brief
Start the experiment without a fixed sample size, re-estimate the required sample size from the outcome data collected so far, and stop once the current sample size exceeds that estimate. Because the stopping rule depends only on the outcome variance and not on the effect estimate, the usual fixed-sample estimator and confidence interval stay valid in large samples (per the paper).
Questions this case answers
- How to decide when to stop a randomized experiment when the variance and effect size needed for sample size planning are unknown in advance and sequential-test corrections cost power
- How to run an A/B test when the required sample size cannot be estimated reliably before it starts?
- How to stop an experiment on its current power estimate without inflating false positives?
- How to check early in a live experiment whether the pre-experiment power analysis was accurate?
Tools and methods
- Fixed-power design — sequential stopping on the estimated required sample size, analyzed with standard fixed-sample inference.
- Power analysis — sample size calculation, repeated during the experiment from current data.
- Sequential testing — the alternative that the design is compared against.
Google or a one-time email code. No password.