Problem
DoorDash runs thousands of experiments in parallel every month, and analysis had no shared standard. Teams ignored clusters in diff-in-diff and switchback analysis, which gives a high false positive rate. Data scientists rebuilt the same methods in every team, and know-how from teams running complex experiments did not spread. Analysis could not keep pace with the number of experiments.
Example from the article: many teams did not account for clusters in diff-in-diff and switchback analysis, which led to a high false positive rate.
Solution in brief
DoorDash built Dash-AB, one statistics library that takes a JSON config with columns and experiment settings. It validates the data, picks a variance method, applies variance reduction, and runs a fixed-horizon or sequential test. The experimentation platform Curie sends data and config to it and shows the results.
Questions this case answers
- How to make every team analyze randomized experiments with the same statistically correct methods when each team has built its own code
- How to compute valid variance, confidence intervals and sample sizes for ratio and quantile metrics on large data without running out of memory
- How to standardize statistical analysis of A/B tests across many product teams?
- How to let non-statisticians run correct experiment analysis from a short config?
Tools and methods
- Dash-AB — in-house central library for experiment analysis, driven by a JSON config
- Curie — in-house experimentation platform that calls Dash-AB
- Chi-square test and flicker test — data validation before analysis
- Regression, delta method, bootstrap — three ways to compute variance
- CUPED and CUPAC — variance reduction with pre-experiment metrics or ML predictions as covariates
- mSPRT — sequential test with always valid p-values
- Diff-in-diff pipeline — analysis of region-level experiments
Google or a one-time email code. No password.