Problem
Skyscanner runs hundreds of A/B experiments, and its analysts had to read every result and decide which variant to ship. That became a bottleneck. Teams also used different methods, which made results hard to compare. As a two-sided marketplace (flights, hotels, car hire), Skyscanner also needs growth in one area not to cost another.
Example from the article: a pop-up saying Skyscanner takes no commission. Variant A is control, B and C are two wordings. The primary metric is the share of users who visit an airline or agency site (redirector rate); the guardrail is the share who book.
Solution in brief
Each experiment pre-registers one primary metric plus guardrails. Every metric gets a Bayesian model, and each variant is compared with control by the 90% highest density interval (HDI) of its relative lift. A fixed rule then recommends the variant that beats control on the primary metric, hurts no guardrail, and has the largest lift.
Questions this case answers
- How to automate the choice of the best variant in a multi-variant experiment when a primary metric must improve and several guardrail metrics must not get worse
- How to define a stopping rule for an experiment that balances test duration against the rate of correct decisions
- How to turn A/B test results into a ship or no-ship decision without manual analysis by analysts?
- How to decide between several variants when one improves the primary metric but may hurt a guardrail metric?
Tools and methods
- WISE — in-house Python library for Bayesian analysis and the ship decision
- Dr Jekyll — in-house experimentation platform that calls WISE
- Beta-binomial model — conversion metrics
- Hurdle gamma-exponential model — revenue per user
- Monte Carlo simulation — calibrates stopping thresholds
- NumPy — vectorized posterior sampling
Google or a one-time email code. No password.