Problem
Analysts check A/B results every day and stop the test once it looks significant. This peeking inflates the false positive rate and leads to misleading conclusions. The usual fix is a fixed sample size that must be reached before any decision. Every test then runs to its planned length, even when the effect is obvious or absent. OLX moved from fixed-horizon tests to group sequential testing (GST) to get decisions earlier at lower cost.
Example from the article: in a simulation where both variants come from the same distribution, 1000 new samples are added per day and a t-test runs daily for 30 days; 27.9% of runs show a "significant" difference at a 5% level.
Solution in brief
Plan interim analyses in advance and compare the test statistic at each one with a precomputed boundary. Boundaries are strict early and relax toward the usual threshold at the end, so the overall error stays at . An optional futility boundary stops tests that are not heading to a win.
Questions this case answers
- How to monitor a randomized experiment at predefined interim points and stop it early while keeping the overall type I error at the chosen level
- How to size an experiment that may stop early so that the worst-case sample is known in advance
- How to check A/B test results at interim points without inflating the false positive rate?
- How to stop a test early when the effect is clear?
Tools and methods
- Group sequential testing (GST) — interim analyses at predefined stages with stopping boundaries
- O'Brien-Fleming (OBF) boundary — efficacy boundary, hard to cross early
- Pampallona and Tsiatis (1994) design — efficacy plus futility boundaries, shaped by
- Python package
group-sequential-testing— the author's code for boundaries and average sample size - NumPy, SciPy — simulation and boundary computation
Google or a one-time email code. No password.