Problem
Uber tests most new features in experiments and usually reads them through the average treatment effect (ATE). An average squeezes the whole outcome distribution into one number, so a change that hurts some riders and helps others can show an ATE of zero. Riders ride many times and remember bad experiences more strongly than good ones, so a fatter tail of long waits can hurt the product even when the mean is unchanged. Splitting by location, time of day or weather does not help when the heterogeneity is not explained by observable factors.
Example from the article: a hypothetical matching algorithm gives the same average pick-up time (ETA) as the incumbent, but more very long and more very short ETAs.
Solution in brief
Compare the treatment and control outcome distributions quantile by quantile. The quantile treatment effect (QTE) at level comes from a quantile regression of the outcome on a constant and a treatment indicator, as a linear regression gives the ATE. Plotting QTEs for quantiles 0.01 to 0.99 shows where the change helps or hurts.
Questions this case answers
- How to detect harm concentrated in the tail of a metric distribution when the average effect of a randomized change looks neutral
- How to detect that a new algorithm made the worst outcomes worse when the average treatment effect is zero?
- How to describe how an experiment changed the whole distribution of a metric, not only its mean?
- How to estimate quantile treatment effects with confidence intervals on millions of observations?
Tools and methods
- Quantile treatment effect (QTE) — difference between the same quantile of the treatment and control outcome distributions
- Quantile regression — estimates the QTE and its confidence interval, like linear regression for the ATE
- MM algorithm of Hunter and Lange — solves the non-differentiable quantile regression objective on millions of rows
Google or a one-time email code. No password.