Problem
A company launching or upgrading an experiment program must pick an inference mode. Many platforms ship a Bayesian mode, sold as modern, easy to read and free of multiple-testing and peeking problems. The usual default, a flat prior with a posterior-probability threshold, reproduces the false positive rate of frequentist peeking. Teams then ship features with no real effect while believing the method protected them.
Example from the article: Spotify analyzes all production experiments with frequentist statistics and weighed adding a Bayesian mode before declining.
Solution in brief
"Bayesian A/B testing" is a family of configurations, each defined by a stopping rule, a prior and a likelihood. Start from the guarantees the program needs (error rates, estimation precision, a cost function), then pick the configuration that gives them. A frequentist group sequential test often suffices. Bayes-factor stopping with a calibrated empirical Bayes prior adds shrinkage and false discovery rate (FDR) control, but only if the prior can be maintained.
Questions this case answers
- How to choose a statistical inference configuration (stopping rule and prior) so that an experiment program keeps its error rates bounded under continuous monitoring
- How to decide whether a calibrated historical prior is worth its maintenance cost when estimating many effects across many metrics
- How to choose between Bayesian and frequentist inference for an experimentation program?
- How to keep the false positive rate under control when people peek at A/B test results?
Tools and methods
- Group sequential testing (GST) — frequentist baseline that bounds the false positive rate under peeking
- Bayes-factor stopping — Bayesian stopping rule that bounds the false positive rate under peeking
- Empirical Bayes prior — prior fitted on historical experiments, gives shrinkage and FDR control
- Mixture sequential probability ratio test (mSPRT) — frequentist sequential test equal to Bayes-factor stopping
- Expected-loss stopping — decision-theoretic rule, optimal only when shipping a null variant costs nothing
Google or a one-time email code. No password.