Problem
Spotify Search has several engineering teams, product managers and a product insights team of data scientists. Engineers must run experiments without waiting for the data scientists, but experiment quality, naming and decision logging differ between teams. Many experiments run on the same users at once, and some must not overlap. Decision makers also want to know what all the shipped changes achieved together, and summing individual test effects is only a naive answer.
Example from the article: in 2021 the Search team started with a search-specific metric and templates, and ended the year with a quarterly cumulative experiment and a rule that every change ships gradually under monitoring.
Solution in brief
Build experimentation practice in three steps: set a quality standard for each experiment, coordinate experiments with each other, then measure total impact with a quarterly holdback group. Data scientists start by doing setup and review work for teams, then hand it over to experimentation champions inside each team.
Questions this case answers
- How to raise the quality and consistency of experiments across several semi-independent product teams without making data scientists a bottleneck
- How to estimate the cumulative causal effect of all product changes shipped over a reporting period
- How to make a product team with engineers, PMs and few data scientists run experiments to a consistent quality standard?
- How to coordinate many parallel experiments so that no user is in two conflicting tests?
Tools and methods
- Holdback (global control) group — users kept out of all product changes for a quarter, used to estimate the total effect
- Mutually exclusive experiments — platform coordination so one user is in only one of the conflicting tests
- Test setup and specification templates — pre-filled parameters, naming, hypothesis, metrics, decision rules
- Experimentation champion — one person per team who spreads practice from the data science team
Google or a one-time email code. No password.