WISE: Skyscanner’s Bayesian AB experimentation library and decision engine

Dhanush Kishore, Jose Parreño, May Alexander, Robert Shepherd25 Apr 2024

Problem

Skyscanner runs hundreds of A/B experiments, and its analysts had to read every result and decide which variant to ship. That became a bottleneck. Teams also used different methods, which made results hard to compare. As a two-sided marketplace (flights, hotels, car hire), Skyscanner also needs growth in one area not to cost another.

Example from the article: a pop-up saying Skyscanner takes no commission. Variant A is control, B and C are two wordings. The primary metric is the share of users who visit an airline or agency site (redirector rate); the guardrail is the share who book.

Solution in brief

Each experiment pre-registers one primary metric plus guardrails. Every metric gets a Bayesian model, and each variant is compared with control by the 90% highest density interval (HDI) of its relative lift. A fixed rule then recommends the variant that beats control on the primary metric, hurts no guardrail, and has the largest lift.

Questions this case answers
  • How to automate the choice of the best variant in a multi-variant experiment when a primary metric must improve and several guardrail metrics must not get worse
  • How to define a stopping rule for an experiment that balances test duration against the rate of correct decisions
  • How to turn A/B test results into a ship or no-ship decision without manual analysis by analysts?
  • How to decide between several variants when one improves the primary metric but may hurt a guardrail metric?

Tools and methods

  • WISE — in-house Python library for Bayesian analysis and the ship decision
  • Dr Jekyll — in-house experimentation platform that calls WISE
  • Beta-binomial model — conversion metrics
  • Hurdle gamma-exponential model — revenue per user
  • Monte Carlo simulation — calibrates stopping thresholds
  • NumPy — vectorized posterior sampling
Continue reading — it's free

Google or a one-time email code. No password.