A Bayesian Approach to A/B Tests: A Conversion Test Example

Kristina Lukyanova27 May 2023

Problem

Teams running conversion A/B tests get a p-value from the frequentist test. It only says whether a difference is statistically significant. It does not say how likely one variant is to be better, or how much conversion or money a wrong choice costs, so business stakeholders find it hard to use. The test also needs a sample size and duration fixed in advance, which a team with little traffic may be unable to reach.

Example from the article: a simulated 4-week conversion test, 30 thousand observations added to each group weekly. The probability that one group has higher conversion grows from 0.77 to 0.93.

Solution in brief

Treat each group's conversion as a distribution, not a point estimate. Update a prior with the observed data to get a posterior, then decide using the probability that one group is better and the expected loss from choosing it. The test can be read at any moment; the dynamics of the criteria show how stable the result is.

Questions this case answers
  • How to decide between two variants of a binary-outcome test from the probability that one is better and the expected loss of a wrong choice instead of a p-value
  • How to stop a binary-outcome test early when the fixed sample size cannot be reached with little traffic
  • How to make A/B test decisions from the probability that one variant is better instead of a p-value
  • How to estimate the expected conversion loss of shipping the wrong variant

Tools and methods

  • Beta-binomial model — beta prior for conversion; the posterior stays beta (conjugate case)
  • Probability that one group is better — decision criterion computed from the joint posterior
  • Expected loss (risk) — how much conversion is lost if the chosen group is actually worse
  • GrowthBook Python library — ready implementation for binary and continuous metrics
Continue reading — it's free

Google or a one-time email code. No password.