Milestones on our Journey to Standardize Experimentation at The New York Times

Kathy Yang26 Mar 2024

Problem

At The New York Times many teams run experiments on very different products, from Wordle to the Audio app to backend migrations. Each team built its own methods and pipelines, so results were hard to compare and analysts maintained their own tooling. Centralizing has a price too, because the local methods fit specific needs. The question is how much to standardize.

Example from the article: a team that did not consistently compute confidence intervals or the minimum detectable effect (MDE) found it much harder to quantify uncertainty before the platform reported them.

Solution in brief

The NYT standardizes only a short list of essentials on its platform ABRA (A/B Reporting and Allocation architecture): governed metrics, daily t-tests with standard outputs, and two extra calculations (scaled impact, log-in rate check). Metric definitions stay with analysts outside the platform. Needs the platform does not cover go to documented off-platform methods.

Questions this case answers
  • How to standardize experiment analysis across many independent teams when each has built its own methods and metric definitions
  • How to compare experiment effects measured on populations of different size and eligibility
  • How to standardize A/B test analysis across many product teams without killing their flexibility?
  • How to split ownership of metric definitions between an experimentation platform and analysts?

Tools and methods

  • Metric governance split — platform team owns exposure and assignment; analysts own metrics
  • Daily t-test report — estimate, confidence interval, p-value, MDE per metric
  • Scaled impact — effect as an estimated global change, comparable across populations
  • Log-in rate check — flags metrics distorted by users switching between logged-out and logged-in
  • Bootstrap pipeline — confidence intervals for quantile metrics
  • Non-inferiority test — treatment is not worse than control by more than a margin
Continue reading — it's free

Google or a one-time email code. No password.