Meet Dash-AB: The Statistics Engine of Experimentation at DoorDash

Caixia Huang, Yixin Tang24 May 2022

Problem

DoorDash runs thousands of experiments in parallel every month, and analysis had no shared standard. Teams ignored clusters in diff-in-diff and switchback analysis, which gives a high false positive rate. Data scientists rebuilt the same methods in every team, and know-how from teams running complex experiments did not spread. Analysis could not keep pace with the number of experiments.

Example from the article: many teams did not account for clusters in diff-in-diff and switchback analysis, which led to a high false positive rate.

Solution in brief

DoorDash built Dash-AB, one statistics library that takes a JSON config with columns and experiment settings. It validates the data, picks a variance method, applies variance reduction, and runs a fixed-horizon or sequential test. The experimentation platform Curie sends data and config to it and shows the results.

Questions this case answers
  • How to make every team analyze randomized experiments with the same statistically correct methods when each team has built its own code
  • How to compute valid variance, confidence intervals and sample sizes for ratio and quantile metrics on large data without running out of memory
  • How to standardize statistical analysis of A/B tests across many product teams?
  • How to let non-statisticians run correct experiment analysis from a short config?

Tools and methods

  • Dash-AB — in-house central library for experiment analysis, driven by a JSON config
  • Curie — in-house experimentation platform that calls Dash-AB
  • Chi-square test and flicker test — data validation before analysis
  • Regression, delta method, bootstrap — three ways to compute variance
  • CUPED and CUPAC — variance reduction with pre-experiment metrics or ML predictions as covariates
  • mSPRT — sequential test with always valid p-values
  • Diff-in-diff pipeline — analysis of region-level experiments
Continue reading — it's free

Google or a one-time email code. No password.