Addressing the Challenges of Sample Ratio Mismatch in A/B Testing

Stas Sajin, Michael Zhou, Krishna Gourishetti17 Oct 2023

Problem

Sample ratio mismatch (SRM) means the groups of an A/B test differ in size from what randomization promised, for example 55/45 instead of 50/50. It breaks random assignment, so the metric readout may be invalid. A chi-square check says that something is wrong, not what. At DoorDash, tests with SRM showed twice as many statistically significant metrics as tests without it.

Example from the article: a test split 50/50 reported a 2% revenue gain ($200,000 a week, over $10 million a year). The real split was 49.5/50.5: about 10,000 employees (dogfooding) were all in treatment, and they spend more than ordinary users. Without them the effect was $0.

Solution in brief

Detect SRM with a chi-square test, then find the cause by regressing treatment assignment on attributes recorded before randomization and running a Wald test per attribute. Unlike separate per-segment tests, the regression separates overlapping attributes. With a real-time dashboard, alerts and training, SRM incidents fell 70% in six months.

Questions this case answers
  • How to localize which pre-randomization attribute drives a group-size imbalance when the candidate attributes are correlated with each other
  • How to detect an unequal split between randomized groups within hours of launch at the scale of billions of assignment events a day
  • How to find which user attribute causes a sample ratio mismatch in an A/B test?
  • How to monitor experiment exposures in real time so an imbalance is caught in the first day?

Tools and methods

  • Chi-square test — first check that group sizes match the planned split
  • Regression of assignment on pre-randomization attributes — shows which attributes relate to assignment
  • Wald test (statsmodels) — one p-value per attribute
  • Weighted regression on aggregated counts — same fit on far fewer rows
  • Apache Flink, Kafka, Pinot — real-time exposure counts by variant
Continue reading — it's free

Google or a one-time email code. No password.