Problem
Sequential tests assume each unit enters the analysis once, with a final value. Faster pipelines break this: users are measured hourly or daily, so their results still change at each look. A metric built from all data so far then drifts between looks, and the test, which expects a fixed target, loses its false positive guarantee. A false positive rate (FPR) above the nominal 0.05 means acting on noise.
Example from the article: Spotify's metric "average minutes played over a week" is built from daily or hourly measurements. Consumption measured in the first five seconds shows nothing, since users have had no time to change behavior.
Solution in brief
Treat measurement, metric, estimand (the effect estimated), estimator and test as separate choices. A cohort-based metric (fixed window after exposure) keeps each unit's value fixed and suits standard sequential tests, at a cost in time or data. An open-ended metric (everything observed so far) uses all data but needs a test that accounts for within-unit covariance. Check any such test by Monte Carlo simulation.
Questions this case answers
- How to keep the false positive rate fixed under repeated interim analyses when each unit is measured many times and its measurement keeps changing between analyses
- How to keep the false positive rate under control when I monitor an A/B test with repeated measurements per user?
- How to choose between a cohort-based and an open-ended metric when I want early readouts?
- How to check by simulation that my sequential test is valid for longitudinal data?
Tools and methods
- Group sequential test (GST) — the standard sequential test whose validity is checked.
- Cohort-based metric — one fixed value per unit over a fixed window since exposure.
- Open-ended metric — a per-unit aggregate over all measurements so far.
- Monte Carlo simulation — estimates the FPR of a procedure under no true effect.
Google or a one-time email code. No password.