Problem
Teams with more ideas than experiment capacity usually test at a two-sided significance level of 0.05 with 80% power. That treats a wrongly shipped change as four times worse than a missed good one. The result is fewer tests, slower product feedback, and a process that small teams with little traffic cannot afford. Loosening the rules alone invites p-hacking, inconsistent metrics and wins that do not replicate.
Example from the article: a growth team ships a promotion with a primary p-value below 0.01 and ignores cost per user rising to $15 against a $10 limit, because the guardrail p-value is 0.2.
Solution in brief
Treat the threshold as a reward-maximization choice: when ideas outnumber capacity, accept a higher false positive rate for more experiments (one-sided tests, one early look). Get trust from standardization instead: shared metric definitions, guardrails that cannot be switched off, pre-registered designs and a visible history of launch decisions.
Questions this case answers
- How to choose error-rate thresholds for a testing program by the expected reward of shipped decisions rather than by a fixed false positive rate
- How to keep many decentralized teams from cancelling each other's gains when each reads its own primary and guardrail metrics
- How to decide how strict an A/B testing process should be when there are more ideas than experiment capacity?
- How to stop teams from shipping wins on a primary metric while a guardrail metric quietly gets worse?
Tools and methods
- One-sided test with one interim look — high-velocity setup: threshold 0.05, stop early on a wrong-direction effect
- Variance reduction, higher alpha, sequential testing — velocity levers when tests run too long
- Pre-registration — fix design and analysis model before launch
- Standardized metrics and always-on guardrails — stop metric arbitrage between teams
- Monte Carlo simulation — compare two decision policies on one stream of ideas
Google or a one-time email code. No password.