Problem
A company with hundreds of teams running A/B tests needs a measure of how well the experimentation program works. Win rate (the share of experiments with a winning treatment) is too narrow. In a mature product most experiments end neutral or negative, and a 0% win rate can still carry large value if tests catch regressions. Without a better measure, nobody can tell whether failed tests come from bad setups, too little data or weak ideas.
Example from the article: in the two Spotify orgs that run about 80% of the company's experiments, the win rate is about 12%, yet about 64% of experiments still produce a usable answer.
Solution in brief
Count an experiment as an Experiment with Learning (EwL) when it is valid and decision-ready: it shipped a win, detected a regression, or was neutral but powered on every relevant metric. The learning rate is the share of launched experiments that meet this bar. The other outcomes show where teams or the platform need work.
Questions this case answers
- How to measure the quality of an experimentation program when most tests do not produce a winner and a win rate undervalues risk management
- How to diagnose, from outcome categories alone, whether low experiment yield comes from invalid setups, insufficient power or early stopping
- How to measure the success of an experimentation program by what it teaches rather than by its win rate?
- How to find out why a team's experiments fail to inform decisions: invalid setups, underpowered tests or early aborts?
Tools and methods
- Experiments with Learning (EwL) rate — share of launched experiments that are valid and decision-ready.
- Outcome classification — ship, abort due to regression, powered without recommendation, unpowered, aborted, failed health checks.
- Health checks and power analysis — decide validity and whether a neutral result is informative.
- Confidence — Spotify's experimentation platform that computes the classification.
Google or a one-time email code. No password.