Problem
Deliveroo runs hundreds of experiments a year, from UI tweaks to algorithm changes, using A/B tests, interleaving, multi-armed bandits and switchbacks. Tooling alone does not keep quality up. Teams that test without a clear hypothesis learn nothing. Teams that pick the rollout rule after seeing results bend the conclusion toward their own idea. The company also wants non-analysts to launch experiments, so the shared rules have to be explicit.
Example from the article: a hypothesis "Another carousel will increase conversion" is rated bad. A good one says an "Order again" carousel cuts search time, because customers find past partners and meals faster, and so raises conversion.
Solution in brief
Deliveroo set five principles: a hypothesis and success criteria before every test, self-service experiments on a shared platform, consistent communication of uncertainty, a value-versus-cost check before testing, and experiments as input to decisions rather than the decision itself.
Questions this case answers
- How to keep experiment quality high when the pool of people running experiments grows beyond specialist analysts
- How to decide up front which changes deserve a controlled test and which should ship without one
- How to run hundreds of experiments a year while non-analysts also launch them, without losing quality?
- How to stop teams from choosing the rollout rule after they have seen the results?
Tools and methods
- A/B test — the default experiment type, run on an in-house experimentation platform.
- Hypothesis and rollout-rule template — written before launch, with the rollout rule fixed up front.
- Value and cost framework ("When to experiment?") — decides whether a test is worth running.
- Results style guide — how to word results and conclusions given the uncertainty.
Google or a one-time email code. No password.