Problem
Teams usually read A/B tests through the change in the mean. Averages hide who moved: a slower app load hurts users on the slowest devices, and an engagement feature may lift only users who were already active. Quantiles show these shifts, but their sampling distribution has no simple formula, so the standard answer is a bootstrap. A bootstrap needs thousands of resampled datasets, which is intractable for tests with millions or hundreds of millions of users. Most companies therefore fall back on normal approximations, whose validity depends on the data distribution.
Example from the article: Spotify wants bootstrap confidence intervals for difference-in-quantiles in A/B tests with hundreds of millions of observations.
Solution in brief
For a quantile estimator based on one order statistic, a Poisson bootstrap never needs to build a resampled dataset. It is enough to know which index of the sorted original sample becomes the quantile in a bootstrap sample. That index is well approximated by a draw. Sort each group once, draw binomial indexes, subtract the picked order statistics, and take quantiles of the differences.
Questions this case answers
- How to quantify uncertainty of a difference in quantiles between two randomized groups when the sample is too large for a standard resampling bootstrap
- How to get confidence intervals for a difference in quantiles (median, 20th percentile) in an A/B test with hundreds of millions of users?
- How to run a bootstrap on a large experiment without materializing thousands of resampled datasets?
- How to check whether a product change helped the least engaged or slowest users, not only the average?
Tools and methods
- Poisson bootstrap — approximates resampling by repeating each observation times; the baseline being sped up.
- Binomial index approximation — replaces resampling with draws of order-statistic indexes.
- NumPy and SciPy — sort, binomial draws and quantiles in the Python examples.
- SQL plus Python — SQL returns order statistics, Python computes the binomial indexes.
Google or a one-time email code. No password.