Problem
A team sizes an A/B test with a calculator at 5% significance and 80% power, and still gets "not significant" most of the time: the speaker cites statistics that about 90% of tests are not significant. Managers peek after 100–1000 observations and blame the analyst when a "win" disappears. The sample size itself rests on guesses: the minimum effect often comes from a business wish, the standard deviation from a historical sample.
Example from the article: a metric with conversion 0.05 in control and a true lift of 0.02. The calculator asks for 1,666 users per group, yet the effect is recognized in only 51.6% of simulated tests.
Solution in brief
Treat the A/B test as an OLS regression of the metric on a 0/1 test flag, and check the planned sample by simulation instead of trusting the calculator. Simulation shows how often a real effect is seen and how often noise passes as an effect. Show the manager these probabilities for several sample sizes and let them choose how long to wait.
Questions this case answers
- How to estimate the effect of a binary treatment with regression and control variables, and check by simulation that the planned sample size really gives the planned power
- How to show a decision maker the probability of detecting an effect at several sample sizes when the minimum effect and the variance are only guesses
- How to choose the sample size of an A/B test and check that it really delivers the planned power
- How to explain to a manager why a real effect often comes out not significant
Tools and methods
- OLS regression (statsmodels) — the
testcoefficient is the effect, with standard error, p-value and confidence interval - Power analysis (
TTestIndPower) — sample size from effect size, alpha, power - Bootstrap simulation — repeated splits of historical data to measure real power and false positives
- Post-stratification, CUPED/CUPAC — control variables that raise sensitivity
Google or a one-time email code. No password.