Using Marketplace Marginal Values to Address Interference Bias

Shima Nassiri, Ido Bright27 Jan 2025

Problem

User-split tests are about 90% of Lyft's randomized studies because they have the most power. In a two-sided marketplace they are biased when one side is scarce. If drivers are short, treated riders who get rides take drivers away from control riders, so the control group looks worse and the treatment effect is overestimated. The same happens in driver-split tests when riders are scarce. This breaks the assumption that treatment does not affect control (SUTVA), and the bias can change a launch decision.

Example from the article: a rider-split test of a change that helps more riders complete trips, run while drivers are undersupplied. Treated riders use up drivers that control riders would have had.

Solution in brief

Replace the face value of each rider or driver with its marginal value: the change in the dispatch objective when one more unit of that resource is added. These values are the dual variables (shadow prices) of the matching optimization, solved once per hourly cycle. Comparing summed marginal values instead of raw outcomes between treatment and control reduces the competition-for-supply bias, with no change to randomization.

Questions this case answers
  • How to estimate the average treatment effect in a user-randomized experiment when treated and control units compete for the same limited resource
  • How to correct a user-split experiment for interference without losing statistical power or changing the randomization design
  • How to remove interference bias from rider-split or driver-split experiments in a two-sided marketplace?
  • How to get an unbiased treatment effect without switching to time-split or region-split experiments?

Tools and methods

  • Marketplace Marginal Values (MMV) — shadow-price metrics that replace raw outcomes in experiment reports
  • Linear programming duality — duals of the relaxed dispatch problem give the marginal values
  • CUPED — variance reduction applied to MMV-corrected metrics, as to any other metric
Continue reading — it's free

Google or a one-time email code. No password.