Imbalance detection for healthier experimentation

Stephane Shao, Lyubov Ignatovsky, Julie BeckleyMar 2022

Problem

Teams at a growing marketplace cannot check by hand that every A/B test compares like with like. Setup bugs, pipeline side effects and asymmetrical triggering or logging can leave treatment and control with different populations. Then a metric change cannot be credited to the feature. Flawed data is rarely salvageable and traffic is scarce, so the alert has to come early, while the sample is small.

Example from the article: a treatment that makes the targeted page fail to load for users with unreliable connections, in a test that logs users only after a successful page load, leaves the treatment group over-represented by users with fast connections.

Solution in brief

For each experiment and each pre-treatment segmentation (for example region), build a contingency table of users by variant and segment, and test whether variant and segment are independent. An exact permutation test gives a valid p-value at any sample size, and the Benjamini-Hochberg procedure controls false alerts across all tests. Aggregated counts and direct sampling of tables keep it cheap at hundreds of experiments a day.

Questions this case answers
  • How to detect, for every running randomized experiment, that the composition of the compared groups differs from what random assignment should produce, without manual review and without a flood of false alarms
  • How to detect automatically that the treatment and control groups of an A/B test consist of different kinds of users?
  • How to run a statistical balance check on hundreds of experiments a day without flooding teams with false alerts?
  • How to test group balance reliably while the experiment is still small?

Tools and methods

  • Contingency table test of independence — checks whether the segment distribution is the same in every variant
  • Permutation test with the USP statistic U — exact p-value that holds for small samples
  • Benjamini-Hochberg procedure — keeps the share of false alerts under control across many tests
  • Apache Spark — runs the tests in parallel
Continue reading — it's free

Google or a one-time email code. No password.