Problem
Teams running a complex algorithmic system, such as DoorDash's assignment of deliveries to couriers ("Dashers"), test changes in live experiments. Failed experiments cost real delivery quality and efficiency. Some changes cannot be tested live at all: the sample is too small for statistical power, or the change alters the environment (for example 20% more orders). The team needs to predict outcomes before launch, yet a simulator of such a system is hard to build.
Example from the article: DoorDash segments its service area into "starting points" and dispatches most deliveries with couriers from the same starting point. This partition is built deeply into production, so live tests of new definitions are hard.
Solution in brief
Build a simulator in four steps. Check that the system is predictable enough, price the benefit (fewer failed experiments, faster iteration), reuse production code behind one API, and grow the simulator through validated milestones with simple v0 models. Always run simulated treatment and simulated control, so that simulator bias cancels out.
Questions this case answers
- How to estimate the effect of an algorithm change in a complex system offline, when live experiments are costly and a failed one hurts users
- How to build a simulator of a stateful system whose parts are predictable but whose aggregate behavior is not
- How to predict whether an algorithm change will fail before launching it as a live experiment?
- How to decide whether a complex system is predictable enough to simulate?
Tools and methods
- Simulation of the assignment system — runs simulated treatment and control through a replica of the production service to estimate experiment effects.
- Dagster — orchestrates parallel batch simulation jobs that call the assignment service.
- Geohash Markov model — predicts where idle couriers go, from transition probabilities between geohash cells.
- Switchback experiments — the production analogue that motivates a minimum simulation duration.
Google or a one-time email code. No password.