Problem
An A/B test is the gold standard, and its result usually reaches the business as one verdict: the p-value is below or above a threshold. Analysts and stakeholders both slip into myths about what the p-value means, for example that means a 5% probability of error. Many tests are also underpowered, so a negative result says little and a positive one may be a fluke that will not repeat. The business still wants one number and a yes or no.
Example from the article: in the early 1970s, US tests of right turns on red lights were too weak to detect more pedestrian deaths. They mostly reported "no problem", while more people were dying.
Solution in brief
Keep the p-value, but report it together with the effect size and its range, and check the test's validity through power and its lower bound. The effect range speaks to the business; the power range shows whether the test deserves trust. Agree on thresholds per product.
Questions this case answers
- How to report a randomized experiment result so that a decision maker sees both the size of the effect and the reliability of the test, not only a significance verdict
- How to quantify the validity of a hypothesis test when a binary significant/not-significant decision hides low statistical power
- How to report A/B test results to the business without reducing them to a single p-value threshold?
- How to judge whether an A/B test was valid enough to trust its verdict, beyond p < 0.05?
Tools and methods
- Cohen's — effect size for continuous metrics such as revenue
- Cohen's — effect size for conversion rates, via the arcsine
- Power range — power over the effect range, a validity check of the test
- statsmodels — functions for both effect sizes
Google or a one-time email code. No password.