Problem
Teams that monitor an A/B test and may stop it early need a sequential test. Most assume one observation per unit. In many experiments each user is measured repeatedly and units enter at different times, so at every look some users have many measurements and others one. A test built for the one-observation case then loses its error guarantee and the false positive rate (FPR) inflates. A platform must fix this across thousands of experiments without holding raw data.
Example from the article: Spotify's open-ended metric (per-unit mean of all measurements so far) with a standard group sequential test (GST) gave inflated FPR in the Part 1 simulation.
Solution in brief
Treat the experiment as a longitudinal model, not a metric. Estimate the effect with OLS, WLS or GLS, handle the within-unit dependence, and apply a GST whose statistics have independent increments. This keeps the FPR nominal and the bounds cheap to compute. Pick a model with few summary statistics.
Questions this case answers
- How to stop a randomized experiment early with valid error control when each unit contributes repeated, correlated measurements that arrive over time
- How to choose a treatment-effect estimator for panel data when the full raw data cannot be held in memory and only compressed statistics are available
- How to run a group sequential test when each unit is measured many times and keep the false positive rate at the nominal level?
- How to choose an estimator for a treatment effect on repeated measurements that can be computed from summary statistics at scale?
Tools and methods
- Group sequential test (GST) — multivariate test over interim statistics, bounds from alpha spending
- Longitudinal model (ROB-OLS, ROB-WLS, GLS) — effect estimators for repeated measurements
- Cluster-robust standard errors (CR0) — correct the standard error for within-unit dependence
- Sufficient statistics — keep the platform's input small
- Monte Carlo simulation — check the FPR of the chosen test before using it
Google or a one-time email code. No password.