Problem
Teams often launch an A/B test without knowing how long it has to run. The sample size depends on four inputs: the baseline rate, the smallest effect worth detecting, the confidence level and the power. It also depends on how traffic is split between variants. A test that is too short gives an inconclusive result, and one that is too long ties up traffic.
Example from the article: with a 5% baseline conversion rate and a 5% relative MDE, a 50/50 test needs 244,242 samples in total. At 1,000 searches per day that takes about 244 days, and at 10,000 per day about 24 days.
Solution in brief
Estimate the sample size per variant with the classic two-proportion z-test formula before launch. Feed it the historical baseline rate, a relative MDE, , power 0.80, a two-tailed test and the traffic split. Divide the total sample size by average daily traffic to get the duration.
Questions this case answers
- How to size a randomized experiment on a rate metric so that it can detect a chosen relative effect within the available traffic
- How to convert a required sample size into a run time when baseline rate and daily traffic are known only from history
- How to estimate how many visitors an A/B test needs before launching it?
- How to turn a required sample size into a test duration from historical traffic?
Tools and methods
- Power analysis / MDE — sets the sample size for a chosen minimum detectable effect, confidence level and power.
- Z-test for two proportions — the formula behind the estimate.
- A/B/n test — the formula extends to several variants by sizing each against the control.
- Algolia A/B testing estimator — dashboard section "Estimate Duration" and an API endpoint that run this calculation on the app's own history.
Google or a one-time email code. No password.