Problem
Uber's first experimentation platform (Morpheus, built 7+ years earlier) handled only a narrow set of designs correctly. Small deviations produced incomparable cohorts, so a large share of experiments had fatal problems and were rerun. Treatment groups lived in mobile client code, so any change cost a 2–4 week build-release cycle. Clients failed closed, so apps and backend services went down when the experimentation backend failed.
Example from the article: in a gradual rollout of a 30/70 control/treatment split, rolling out to 10% of users was fine but rolling out to 5% was not, because of how rollout and assignment interacted.
Solution in brief
Uber rebuilt the platform around one concept, the experiment (randomization, treatment plan, logs), and one client-visible concept, the parameter. Randomization is a salted hash into buckets, so any combination of experiments stays comparable. Holdouts, traffic splits and dependent experiments are built from plain experiments through parameter constraints.
Questions this case answers
- How to build an experimentation platform that guarantees comparable treatment groups for any experiment configuration, including gradual rollouts and nested holdouts
- How to decouple experiment definitions from client code so designs change without a client release and clients survive a platform outage
- How to build an A/B platform whose cohorts stay comparable whatever experiment design a user picks?
- How to run hierarchical designs (holdouts, traffic splits, dependent experiments) without special constructs?
Tools and methods
- Parameter — the only thing client code sees; has a safe default
- Salted-hash bucketing with a treatment-group tree — assigns units to buckets and groups
- Treatment plan with context — maps context and group to parameter values
- Parameter constraints — build holdouts, traffic splits, dependent experiments
- SDK exposure logging — logs each access of an overridden parameter
- Logged and logless cohorts — analysis by exposure logs or by independent definitions
- Holdout group — an experiment that sets a parameter
Google or a one-time email code. No password.