Sequential A/B Testing Keeps the World Streaming Netflix Part 2: Counting Processes

Michael Lindon, Chris Sanden, Vache Shirikian, Yanjun Liu, Minal Mishra, Martin Tingley18 Mar 2024

Problem

Netflix rolls new client code out to a small share of devices (a canary test) and watches many count metrics: logins, errors, successful title starts, abnormal shutdowns, call-center contacts. A bug must be caught quickly, because every extra minute disrupts playback for real users. Repeated looks with a fixed-horizon test inflate false alarms, so the test must stay valid under continuous monitoring. Event rates also change over time.

Example from the article: a canary showed fewer successful title starts in treatment. The bug blocked about 60% of treatment devices from starting a stream, and it was detected at the sub-second level.

Solution in brief

Model the events of each group as a time-inhomogeneous Poisson process and test whether both groups share the same intensity function, with no assumption about its shape. The only inputs at any moment are two integers: events seen so far in control and in treatment. The test returns a sequential p-value and a confidence sequence that stay valid at every look.

Questions this case answers
  • How to test continuously, with valid false positive control, whether two streams of timestamped events arrive at different rates when the event rate varies over time
  • How to monitor many count metrics during a staged rollout and stop at the first sign of a regression
  • How to detect as early as possible that a new software release produces fewer successful events (logins, title starts) than the current one?
  • How to detect that errors or crashes accrue faster in a canary than in control without inflating false alarms from repeated looks?

Tools and methods

  • Sequential testing (confidence sequences, sequential p-values) — decide at any moment without inflating false positives
  • Time-inhomogeneous Poisson point process — model of event arrivals with a time-varying rate
  • Canary rollout — send the new release to a small random share of devices and compare with control
  • Sequential p-value script (GitHub gist) — computes the p-value from the two counts
Continue reading — it's free

Google or a one-time email code. No password.