Speed with Rigor: testing smarter with group sequential design

Gabriela Lewenfus5 Aug 2025

Problem

Analysts check A/B results every day and stop the test once it looks significant. This peeking inflates the false positive rate and leads to misleading conclusions. The usual fix is a fixed sample size that must be reached before any decision. Every test then runs to its planned length, even when the effect is obvious or absent. OLX moved from fixed-horizon tests to group sequential testing (GST) to get decisions earlier at lower cost.

Example from the article: in a simulation where both variants come from the same distribution, 1000 new samples are added per day and a t-test runs daily for 30 days; 27.9% of runs show a "significant" difference at a 5% level.

Solution in brief

Plan K interim analyses in advance and compare the test statistic at each one with a precomputed boundary. Boundaries are strict early and relax toward the usual threshold at the end, so the overall error stays at α. An optional futility boundary stops tests that are not heading to a win.

Questions this case answers
  • How to monitor a randomized experiment at predefined interim points and stop it early while keeping the overall type I error at the chosen level
  • How to size an experiment that may stop early so that the worst-case sample is known in advance
  • How to check A/B test results at interim points without inflating the false positive rate?
  • How to stop a test early when the effect is clear?

Tools and methods

  • Group sequential testing (GST) — interim analyses at predefined stages with stopping boundaries
  • O'Brien-Fleming (OBF) boundary — efficacy boundary, hard to cross early
  • Pampallona and Tsiatis (1994) design — efficacy plus futility boundaries, shaped by Δ
  • Python package group-sequential-testing — the author's code for boundaries and average sample size
  • NumPy, SciPy — simulation and boundary computation
Continue reading — it's free

Google or a one-time email code. No password.