Why Spotify Is Not Using Bayesian A/B Testing

Mattias Frånberg, Mårten SchultzbergSep 2026

Problem

A company launching or upgrading an experiment program must pick an inference mode. Many platforms ship a Bayesian mode, sold as modern, easy to read and free of multiple-testing and peeking problems. The usual default, a flat prior with a posterior-probability threshold, reproduces the false positive rate of frequentist peeking. Teams then ship features with no real effect while believing the method protected them.

Example from the article: Spotify analyzes all production experiments with frequentist statistics and weighed adding a Bayesian mode before declining.

Solution in brief

"Bayesian A/B testing" is a family of configurations, each defined by a stopping rule, a prior and a likelihood. Start from the guarantees the program needs (error rates, estimation precision, a cost function), then pick the configuration that gives them. A frequentist group sequential test often suffices. Bayes-factor stopping with a calibrated empirical Bayes prior adds shrinkage and false discovery rate (FDR) control, but only if the prior can be maintained.

Questions this case answers
  • How to choose a statistical inference configuration (stopping rule and prior) so that an experiment program keeps its error rates bounded under continuous monitoring
  • How to decide whether a calibrated historical prior is worth its maintenance cost when estimating many effects across many metrics
  • How to choose between Bayesian and frequentist inference for an experimentation program?
  • How to keep the false positive rate under control when people peek at A/B test results?

Tools and methods

  • Group sequential testing (GST) — frequentist baseline that bounds the false positive rate under peeking
  • Bayes-factor stopping — Bayesian stopping rule that bounds the false positive rate under peeking
  • Empirical Bayes prior — prior fitted on historical experiments, gives shrinkage and FDR control
  • Mixture sequential probability ratio test (mSPRT) — frequentist sequential test equal to Bayes-factor stopping
  • Expected-loss stopping — decision-theoretic rule, optimal only when shipping a null variant costs nothing
Continue reading — it's free

Google or a one-time email code. No password.