Beyond A/B Testing: Using Surrogacy and Region-Splits to Measure Long-Term Effects in Marketplaces

Amber Wang, Yoonji Kim25 Mar 2026

Problem

A ride-hailing marketplace must decide how to split budget between rider prices and driver incentives. Long-term effects of these decisions tend to outweigh the short-term ones: charging riders more earns profit today but loses riders and drivers later. Part of the long-term effect is direct (an incentivized driver returns more often). The harder part is market-mediated: more driver hours cut rider wait and surge, but also leave drivers idle and less willing to return. No single experiment verifies it.

Example from the article: raise driver incentive spend by x% in week 1, then estimate how rider return rates and future driver hours change through shorter waits and higher idleness.

Solution in brief

Assume market-mediated effects pass fully through a few negative user experiences (long waits, high surge, cancellations, idleness). Step 1 estimates how a policy shifts those experiences. Step 2 estimates how they change future behavior. Direct effects are added, and the total is checked in region-split experiments.

Questions this case answers
  • How to estimate the long-term effect of a policy change when the effect passes through other participants of a two-sided market and a long randomized test is too slow
  • How to validate a chain of observational causal estimates when each link can be checked only by a different imperfect experiment
  • How to estimate the long-term effect of a price or incentive change in a two-sided marketplace without running a long experiment?
  • How to separate direct long-term effects from effects that pass through the market (wait times, surge, driver idleness)?

Tools and methods

  • Residualized regression — policy effect on negative experiences, measured as deviation from the time-of-week norm
  • AIPW (doubly robust estimator) — effect of negative experiences on future rides, retention and driver hours
  • Surrogacy index — scales a short-term experience shift into a long-term effect
  • Switch-back, user-split and region-split experiments — validate each step and the final forecast
Continue reading — it's free

Google or a one-time email code. No password.