Problem
In a classic A/B test the sample size and the traffic split are fixed in advance. Even when one arm clearly wins early, the team keeps serving the weaker arms until the planned end date. This slows experimentation and makes it expensive to test many ideas, for example a long list of parameter values for a machine learning model. Teams want an automated early signal that an arm is unlikely to succeed.
Example from the article: DoorDash teams with a long list of ideas, such as parameter values of a machine learning model, have to wait for each fixed-horizon test to finish before they can narrow the options.
Solution in brief
DoorDash built a multi-armed bandit (MAB) platform on top of its experimentation system. A daily batch job runs Thompson sampling and writes the recommended traffic share for each arm. The main change from the textbook version is that the model keeps priors on the treatment effect against control, not on raw metric values, which removes a Simpson's paradox bias.
Questions this case answers
- How to reallocate experiment traffic toward better variants during the test when metrics arrive in daily batches and the environment is non-stationary
- How to narrow a long list of candidate variants quickly without paying the opportunity cost of a fixed-allocation test
- How to stop sending traffic to losing variants before a fixed-horizon A/B test ends?
- How to screen many candidate ideas or parameter values cheaply before a full A/B test?
Tools and methods
- Thompson sampling — Bayesian policy that picks traffic shares from posterior draws
- Beta and normal priors — reward models for binomial and normal metrics
- Reward worker — daily batch job that runs the Bayesian update and computes allocation
- MAB service — stores configs and reward distributions, serves current allocation to downstream apps
- Treatment-effect priors — posteriors defined relative to the control arm
Google or a one-time email code. No password.