Balancing Velocity and Confidence in Experimentation

Stas Sajin15 Nov 2022

Problem

Teams with more ideas than experiment capacity usually test at a two-sided significance level of 0.05 with 80% power. That treats a wrongly shipped change as four times worse than a missed good one. The result is fewer tests, slower product feedback, and a process that small teams with little traffic cannot afford. Loosening the rules alone invites p-hacking, inconsistent metrics and wins that do not replicate.

Example from the article: a growth team ships a promotion with a primary p-value below 0.01 and ignores cost per user rising to $15 against a $10 limit, because the guardrail p-value is 0.2.

Solution in brief

Treat the threshold as a reward-maximization choice: when ideas outnumber capacity, accept a higher false positive rate for more experiments (one-sided tests, one early look). Get trust from standardization instead: shared metric definitions, guardrails that cannot be switched off, pre-registered designs and a visible history of launch decisions.

Questions this case answers
  • How to choose error-rate thresholds for a testing program by the expected reward of shipped decisions rather than by a fixed false positive rate
  • How to keep many decentralized teams from cancelling each other's gains when each reads its own primary and guardrail metrics
  • How to decide how strict an A/B testing process should be when there are more ideas than experiment capacity?
  • How to stop teams from shipping wins on a primary metric while a guardrail metric quietly gets worse?

Tools and methods

  • One-sided test with one interim look — high-velocity setup: threshold 0.05, stop early on a wrong-direction effect
  • Variance reduction, higher alpha, sequential testing — velocity levers when tests run too long
  • Pre-registration — fix design and analysis model before launch
  • Standardized metrics and always-on guardrails — stop metric arbitrage between teams
  • Monte Carlo simulation — compare two decision policies on one stream of ideas
Continue reading — it's free

Google or a one-time email code. No password.