Method
Bayesian A/B test
Experiment analysis with posteriors and probability-to-beat-control instead of p-values.
7 cases from 7 companies apply Bayesian A/B test. 1 was checked in an online A/B test.
Typical tasks: Avoid false wins in A/B tests, Get experiment results faster, Predict long-term effect from short tests, Experiment when users affect each other and 1 more. Companies: Spotify, Sber, Khan Academy, Netflix and 3 more. Often combined with: Bootstrap, FDR correction, Guardrails, Sequential testing, t-test and 19 more.
| Why Spotify Is Not Using Bayesian A/B Testing | media-streaming | 2026-09 | |
| From Eight Hours to Eight Minutes: Automating A/B Test Analysis on Databricks | gaming | 2026-06 | |
| Why Reports Mislead, or How to Measure the Effect of an LLM Feature | general | 2026-05 | |
| How Khan Academy Is Building a Better AI Tutor: Our Most Recent Learnings | edtech | 2026-05 | |
| WISE: Skyscanner’s Bayesian AB experimentation library and decision engineSK | SK | travel | 2024-04 |
| A Bayesian Approach to A/B Tests: A Conversion Test ExampleGL | GL | general | 2023-05 |
| Experimentation is a major focus of Data Science across Netflix | media-streaming | 2022-01 |