Measuring Marketing Effectiveness in a Cookie-Less World

Arik Motskin, Michael Haugan, Parth Aidasani29 Apr 2024

Problem

A growth team that spends heavily on paid media needs to know what each channel adds beyond what would have happened anyway. Last-touch and multi-touch attribution under-credit view-based, top-of-funnel channels such as video, and over-credit search: a user may see YouTube ads, later search for the brand and convert, and search gets the credit. The end of third-party cookies makes tracking worse. Budget then goes to the wrong channels.

Example from the article: Grammarly ran a YouTube go-dark experiment and measured the incremental effect on new active users.

Solution in brief

Turn the channel off in a set of geographic markets (the test set) and keep it on in the rest (the control set). Pick the two sets so a time-series model trained on history predicts the test KPI accurately. After the intervention, compare actual test performance with a synthetic baseline and read the difference as incremental lift. Size and length come from a simulation-based power analysis.

Questions this case answers
  • How to estimate the causal effect of a marketing channel on acquisition when user-level tracking is unavailable and only region-level randomization is possible
  • How to choose balanced region groups and the experiment size and length when the number of units is small and switching a channel off has a business cost
  • How to measure the incremental effect of a top-of-funnel ad channel when click-based attribution credits the wrong channel?
  • How to choose a balanced test/control split of geographic markets for a go-dark experiment?

Tools and methods

  • Geo go-dark experiment — regions as experiment units, spend off in test regions.
  • Time-based regression (TBR) with Bayesian structural time series — synthetic baseline and daily intervals.
  • Split generation: clustering and randomization — candidate test/control splits.
  • Power analysis by simulation — choose share of markets and duration.
  • CausalImpact — considered, not used: better suited to observational data.
Continue reading — it's free

Google or a one-time email code. No password.