Experimentation is a major focus of Data Science across Netflix

Netflix Technology Blog25 Jan 2022

Problem

Each business area at Netflix breaks the textbook A/B test in its own way. Advertising tests are low-powered and every tested ad costs money, so a wrongly promoted treatment wastes budget. Payments teams need results fast because traffic and time are limited. Partnership launches and network changes often cannot be randomized at all. The post names the obstacle and the method for each area.

Example from the article: in growth advertising a statistically significant winner from a low-powered test can have the wrong sign (Type-S error) or an exaggerated size (Type-M error), which skews future budget levels.

Solution in brief

Match the method to the obstacle. Use a Bayesian prior built from past tests to shrink noisy estimates, sequential or Bayesian stopping to end bad arms early, and quasi-experimental methods when randomization is impossible. Put shared metric definitions, assignment and analysis into one platform that data scientists can extend.

Questions this case answers
  • How to make cost-sensitive decisions from experiments whose statistical power is too low to trust a significant result at face value
  • How to end losing experiment arms early while keeping error rates valid
  • How to estimate the causal effect of an experience when users self-select into it or it is rolled out to a whole cluster at once
  • How to get reliable ad-spend decisions from low-powered experiments?

Tools and methods

  • Bayesian analysis with a prior from past tests — shrinks estimated effects and intervals to reduce Type-S and Type-M errors
  • Group sequential testing, Gaussian Bayesian inference, adaptive testing — faster stopping decisions
  • Difference-in-differences, inverse propensity scores, doubly robust estimators, instrumental variables — causal effects under self-selection
  • Synthetic controls and structural models — effects when only aggregate data exist
  • Bootstrap for compressed data, quantile treatment effects — skewed streaming metrics
  • Bandits — continuous learning for messaging and artwork selection
Continue reading — it's free

Google or a one-time email code. No password.