Problem
The Big Data Media team at MTS tests personalized recommendations in KION, an online cinema: a single "Especially for you" shelf and the ranking of the whole feed. Monitoring invites the team to look at a running test, but a classical test cannot be stopped before the planned sample is reached. Stopping on the first significant day means fooling yourself. The team also needed to screen 10 or more models against one control without splitting traffic into many long tests.
Example from the article: in an A/A simulation, a t-test checked every day gave 134 false positives out of 1000 runs at , against 55 for a single check on the last day.
Solution in brief
Use a sequential test from the mSPRT family. It turns the likelihood ratio into an always-valid p-value that can be compared with on any day. The team checked the false positive rate on its own data and used the test to screen models in one week, with a correction for the number of comparisons.
Questions this case answers
- How to look at a running online test daily and stop it early without inflating the false positive rate
- How to screen many variants against one control in a single sequential test with a correction for multiple comparisons
- How to peek at an A/B test of a recommendation model without inflating false positives
- How to stop an A/B test earlier than the planned sample size
Tools and methods
- mSPRT (mixture sequential probability ratio test) — a sequential test with a mixed distribution under the alternative; gives always-valid p-values
- Wald SPRT — the classical ancestor, with a fixed alternative effect and a corridor of boundaries
- A/A simulation on historical data — 1000 shuffles to measure false positives
- Correction for multiple comparisons — for several models against one control
Google or a one-time email code. No password.