Challenges in Experimentation

John Kirn19 Apr 2022

Problem

Lyft tests every product change, but a standard user-level A/B test fails in a ride-sharing marketplace. Riders and drivers share one pool of supply, so one user's treatment changes what another user gets. The independence assumption between groups breaks. Weather, events and outages shift behavior fast. With many thousands of experiments a year, coordination gets harder, and every extra metric in a test raises the false positive rate.

Example from the article: if riders in treatment click "Request Now" more often than riders in control click "Request", the treatment riders match with nearby drivers first, and control riders lose out.

Solution in brief

Lyft picks the randomization unit to fit the interference: users, time windows, regions with a synthetic control, sessions or hardware. Around that sits process: a pre-registered hypothesis, Benjamini-Hochberg correction for many primary metrics, an experiment scheduler and a decision log. Adaptive experiments are still being built.

Questions this case answers
  • How to estimate the causal effect of a change when units interfere and user-level randomization would bias the result
  • How to keep the false discovery rate low when each experiment reports many primary metrics
  • How to keep an experimentation program reliable as teams, randomization units and the number of tests grow
  • How to measure the effect of a change when users in one test group affect users in the other?

Tools and methods

  • User split A/B test — the most common type
  • Time split test — one experience per geography and time window; for pricing, ETAs, routing, mapping
  • Region split with synthetic control — treatment by geography, counterfactual from pre-test trends
  • Residualization — variance reduction for time splits, being implemented
  • Benjamini-Hochberg correction — p-value adjustment for several primary metrics
  • Guided hypothesis workflow — pre-registers treatment, population and metrics
  • Contextual bandits, parameter tuning — adaptive experiments, in development
Continue reading — it's free

Google or a one-time email code. No password.