Supercharging A/B Testing at Uber

Sergey Gitlin, Krishna Puttaswamy, Luke Duncan21 Jul 2022

Problem

Uber's first experimentation platform (Morpheus, built 7+ years earlier) handled only a narrow set of designs correctly. Small deviations produced incomparable cohorts, so a large share of experiments had fatal problems and were rerun. Treatment groups lived in mobile client code, so any change cost a 2–4 week build-release cycle. Clients failed closed, so apps and backend services went down when the experimentation backend failed.

Example from the article: in a gradual rollout of a 30/70 control/treatment split, rolling out to 10% of users was fine but rolling out to 5% was not, because of how rollout and assignment interacted.

Solution in brief

Uber rebuilt the platform around one concept, the experiment (randomization, treatment plan, logs), and one client-visible concept, the parameter. Randomization is a salted hash into buckets, so any combination of experiments stays comparable. Holdouts, traffic splits and dependent experiments are built from plain experiments through parameter constraints.

Questions this case answers
  • How to build an experimentation platform that guarantees comparable treatment groups for any experiment configuration, including gradual rollouts and nested holdouts
  • How to decouple experiment definitions from client code so designs change without a client release and clients survive a platform outage
  • How to build an A/B platform whose cohorts stay comparable whatever experiment design a user picks?
  • How to run hierarchical designs (holdouts, traffic splits, dependent experiments) without special constructs?

Tools and methods

  • Parameter — the only thing client code sees; has a safe default
  • Salted-hash bucketing with a treatment-group tree — assigns units to buckets and groups
  • Treatment plan with context — maps context and group to parameter values
  • Parameter constraints — build holdouts, traffic splits, dependent experiments
  • SDK exposure logging — logs each access of an overridden parameter
  • Logged and logless cohorts — analysis by exposure logs or by independent definitions
  • Holdout group — an experiment that sets a parameter
Continue reading — it's free

Google or a one-time email code. No password.