Problem
Chrome plans to phase out third-party cookies (3PCs). In a market test overseen by the UK regulator (CMA), advertising platforms (DSPs) try the replacement APIs on a small slice of Chrome traffic. The slice is fixed from outside: 0.75% in Mode B, up to 8.25% if Mode A is also used. A platform cannot tell in advance whether it can separate a true drop in conversions per dollar from noise in the allowed time. A test that is too small or short shows "no significant change" even if performance fell.
Example from the article: Criteo asked what drop in conversions per dollar it could confidently measure in eight weeks, and what a DSP 100 times smaller than Criteo could measure.
Solution in brief
Model the test statistic (the ratio of conversions per dollar, treatment to control) with a Beta prime distribution. Fit its parameters on 1000 simulated A/A splits of your own users, take the 5th percentile as the critical value, then find the true drop that would be caught with 80% power. This drop is the Minimum Detectable Downlift (MDD). One elasticity parameter scales the noise to smaller platforms.
Questions this case answers
- How to compute the minimum detectable drop in a spend-normalised conversion ratio when the treatment is a small, externally fixed share of traffic and the test length is capped
- How to extrapolate the measurement noise of a large participant to smaller participants of the same market when only their spend differs
- How to find the smallest drop in conversions per dollar that a market test can detect in a fixed number of weeks?
- How to size the treatment group and test duration when a platform change is tested on a small share of traffic?
Tools and methods
- Power analysis — MDD for a given treatment share and horizon
- A/A test — 1000 random splits of existing users to measure pure noise
- Hypothesis testing — one-sided test, 95% significance, 80% power
Google or a one-time email code. No password.