Sequential A/B Testing Keeps the World Streaming Netflix. Part 1: Continuous Data

Michael Lindon, Chris Sanden, Vache Shirikian, Yanjun Liu, Minal Mishra, Martin Tingley19 Mar 2024

Problem

Netflix releases client software through canary tests: a randomized A/B test where a small group gets the new version. The test must stop a bad release fast, so the team checks it all the time. Repeated checks with an ordinary fixed-sample test break its false positive guarantee and send developers after bugs that do not exist. The metric is also a whole distribution: the mean can stay flat while the slow tail gets worse.

Example from the article: play-delay, the time from pressing play to playback starting. Any increase is a regression, including one that only hits members on slow connections.

Solution in brief

Replace the fixed-sample test with an anytime-valid one. Build time-uniform confidence bands on the quantile functions of both arms and on their difference, and stop as soon as the band on the difference excludes zero at any quantile. The error guarantee holds at every moment, so the test can run after each data point.

Questions this case answers
  • How to test continuously for any difference between two metric distributions while keeping the false positive rate fixed under repeated looks
  • How to detect large regressions early and small ones eventually when the sample size cannot be fixed in advance
  • How to monitor a canary test continuously and stop it the moment the new version is worse, without inflating false alarms?
  • How to test for any difference in a metric's distribution, including the tail, and not only in its mean?

Tools and methods

  • Sequential (anytime-valid) testing — error control that survives continuous peeking
  • Time-uniform confidence bands on quantile functions — one test for any change in the distribution
  • Sequential p-value — a running summary that matches the band
  • Canary test — randomized check of a new software version on a small user subset
Continue reading — it's free

Google or a one-time email code. No password.