Problem
Sample ratio mismatch (SRM) means the groups of an A/B test differ in size from what randomization promised, for example 55/45 instead of 50/50. It breaks random assignment, so the metric readout may be invalid. A chi-square check says that something is wrong, not what. At DoorDash, tests with SRM showed twice as many statistically significant metrics as tests without it.
Example from the article: a test split 50/50 reported a 2% revenue gain ($200,000 a week, over $10 million a year). The real split was 49.5/50.5: about 10,000 employees (dogfooding) were all in treatment, and they spend more than ordinary users. Without them the effect was $0.
Solution in brief
Detect SRM with a chi-square test, then find the cause by regressing treatment assignment on attributes recorded before randomization and running a Wald test per attribute. Unlike separate per-segment tests, the regression separates overlapping attributes. With a real-time dashboard, alerts and training, SRM incidents fell 70% in six months.
Questions this case answers
- How to localize which pre-randomization attribute drives a group-size imbalance when the candidate attributes are correlated with each other
- How to detect an unequal split between randomized groups within hours of launch at the scale of billions of assignment events a day
- How to find which user attribute causes a sample ratio mismatch in an A/B test?
- How to monitor experiment exposures in real time so an imbalance is caught in the first day?
Tools and methods
- Chi-square test — first check that group sizes match the planned split
- Regression of assignment on pre-randomization attributes — shows which attributes relate to assignment
- Wald test (statsmodels) — one p-value per attribute
- Weighted regression on aggregated counts — same fit on far fewer rows
- Apache Flink, Kafka, Pinot — real-time exposure counts by variant
Google or a one-time email code. No password.