Privacy Sandbox Market Testing: a statistical model for measuring performance impacts

Caio Camilli5 Dec 2023

Problem

Chrome plans to phase out third-party cookies (3PCs). In a market test overseen by the UK regulator (CMA), advertising platforms (DSPs) try the replacement APIs on a small slice of Chrome traffic. The slice is fixed from outside: 0.75% in Mode B, up to 8.25% if Mode A is also used. A platform cannot tell in advance whether it can separate a true drop in conversions per dollar from noise in the allowed time. A test that is too small or short shows "no significant change" even if performance fell.

Example from the article: Criteo asked what drop in conversions per dollar it could confidently measure in eight weeks, and what a DSP 100 times smaller than Criteo could measure.

Solution in brief

Model the test statistic (the ratio of conversions per dollar, treatment to control) with a Beta prime distribution. Fit its parameters on 1000 simulated A/A splits of your own users, take the 5th percentile as the critical value, then find the true drop that would be caught with 80% power. This drop is the Minimum Detectable Downlift (MDD). One elasticity parameter scales the noise to smaller platforms.

Questions this case answers
  • How to compute the minimum detectable drop in a spend-normalised conversion ratio when the treatment is a small, externally fixed share of traffic and the test length is capped
  • How to extrapolate the measurement noise of a large participant to smaller participants of the same market when only their spend differs
  • How to find the smallest drop in conversions per dollar that a market test can detect in a fixed number of weeks?
  • How to size the treatment group and test duration when a platform change is tested on a small share of traffic?

Tools and methods

  • Power analysis — MDD for a given treatment share and horizon
  • A/A test — 1000 random splits of existing users to measure pure noise
  • Hypothesis testing — one-sided test, 95% significance, 80% power
Continue reading — it's free

Google or a one-time email code. No password.