Criteo's Privacy Sandbox Market Testing

Elias Selman18 Jul 2024

Problem

Criteo, an ad-tech company, needed to estimate how much publisher revenue would fall if Chrome removed third-party cookies and advertisers moved to Google's Privacy Sandbox. The test ran inside an industry-wide test initiated by the CMA and Google on a small share of Chrome traffic. Three things threatened the answer. Advertisers adjust budgets when performance changes. Supply partners forwarded group labels for only part of the traffic. And the Google supply platform gained share inside the treated group. A naive readout makes the loss look smaller.

Example from the article: Reading only Google Ad Manager (GAM) traffic showed a 21% revenue drop, while the whole market lost 78%.

Solution in brief

Run the test across all supply at constant advertiser performance, with a per-campaign feedback loop. Recover group labels from three sources, normalize each group to its official share of Chrome traffic, and strip leftover cookie traffic from the treated group. Then compare publisher revenue against the cookie-based control.

Questions this case answers
  • How to estimate the market-level effect of a platform change on a third party's revenue when the platform operator's own channel gains share from competitors in the treated group
  • How to analyse a randomized test when group labels reach the analyst for only part of the traffic and the missing share differs between groups
  • How to measure the market-level effect of a platform change when competing intermediaries react to it?
  • How to keep a test fair when a share of traffic arrives without its group label?

Tools and methods

  • Three-population A/B design — Treatment (Privacy Sandbox, no cookies), Control 1 (cookies), Control 2 (contextual only).
  • Constant-performance feedback loop — per-campaign controller that matches performance of Treatment and Control 2 to Control 1.
  • Label recovery — labels from bid requests, stored labels, labels seen at display time.
  • Population size regularization — scales each group's metrics by its official traffic share.
Continue reading — it's free

Google or a one-time email code. No password.