Problem
Experiments at DoorDash sometimes raised app latency and error rates and, in extreme cases, caused crashes. App-quality metrics were not standardized, so experiment owners did not know what to watch or what to do when a metric moved. Bad rollouts were not paused, and a problematic experiment could reach full rollout unnoticed.
Example from the article: metric and page owners did not know which experiments were hurting their metrics, so finding the cause of a degradation could take several weeks.
Solution in brief
The experimentation platform runs an automatic app-quality analysis on every consumer experiment with a small standard metric set. It uses multiple-testing correction, sequential testing and a per-metric tolerance. Degradation beyond tolerance pauses or blocks the rollout and alerts the owner; smaller regressions go through an approval and repayment process.
Questions this case answers
- How to automatically pause a staged rollout when a standardized guardrail metric degrades, without flooding teams with false alarms
- How to set per-metric tolerance for acceptable regression and tie it to an approval and repayment process
- How to stop experiments that degrade app latency, errors or crashes before they reach full rollout?
- How to give experiment owners a standard set of app-quality metrics so they know what to watch?
Tools and methods
- Composite app-quality metrics — share of "bad" page and action events, checked first to keep false positives and compute low
- Five diagnostic sub-metrics with dimension cuts — used only after a composite metric degrades
- Multiple-testing correction and sequential testing — keep false alarms low without requiring owners to know the statistics
- Per-metric tolerance — a degradation threshold set by the metric owner
- Clawback plan and budget framework — approval rules for accepted short-term regressions
Google or a one-time email code. No password.