Choosing a Sequential Testing Framework — Comparisons and Discussions

Mårten Schultzberg, Sebastian AnkargrenMar 2023

Problem

Teams running online experiments want to watch results while data is still coming in, mainly to catch regressions early. A standard z-test or t-test keeps its false positive rate only if applied once, after data collection ends. Repeated looks ("peeking") give several chances to find a false positive, so the real rate grows far above the intended α. Sequential tests fix this, but the families differ in requirements and there is little advice on choosing.

Example from the article: a z-test built for a 5% false positive rate, applied after each new pair of observations. After two tests the real rate is already close to 10%.

Solution in brief

Choose by two questions: does data arrive in batches or as a stream, and can you estimate the maximum sample size? In Spotify's simulations the group sequential test (GST) has the highest power whenever the sample size can be estimated. Always valid inference (AVI) fits when the sample size is unknown and data is streamed.

Questions this case answers
  • How to monitor a randomized experiment repeatedly during data collection while keeping the false positive rate at the intended level and losing as little power as possible
  • How to choose a sequential testing method when the final sample size may be unknown and data arrives either in batches or as a stream
  • How to choose a sequential test that lets us monitor an A/B test continuously without inflating false positives?
  • How to keep power high when the final sample size of an experiment is uncertain?

Tools and methods

  • Group sequential test (GST) — repeated z-tests with alpha spending; needs the expected sample size.
  • mSPRT and GAVI (always valid inference) — valid at every observation; need a mixing parameter.
  • Bonferroni correction — z-test with α divided by the number of planned analyses.
  • Corrected-alpha approach (Statsig) — reduces but does not bound the inflation.
  • ldbounds R package — computes GST bounds.
Continue reading — it's free

Google or a one-time email code. No password.