Measuring marketing incremental impacts beyond last click attribution

Maggie Zhang, Joyce Chen, Ming Wu14 Jul 2022

Problem

Marketing teams need to know how many conversions a campaign adds beyond organic ones. User-level A/B tests do not work for TV, radio and billboards, and privacy rules complicate them on third-party platforms. The usual fallback is last-click attribution, which gives all credit to the final touchpoint. It ignores earlier exposures and over-credits paid media for conversions that would have come organically. Channel ROI and budget allocation then rest on a wrong picture.

Example from the article: LinkedIn needed to know how much of the app installs reported by a Google Universal App campaign on Android were truly incremental.

Solution in brief

Run the campaign in selected regions only and use the other regions as controls. A Bayesian structural time series (BSTS) model, trained on the pre-campaign period, predicts what the test regions would have done without the campaign. The gap between observed and predicted values is the incremental effect. Before launch, an A/A test and a simulation-based power analysis check that the design can work.

Questions this case answers
  • How to estimate the causal effect of an intervention when units cannot be randomized individually and only aggregated time series per region are available
  • How to design and power a region-level experiment when no closed-form power formula exists
  • How to measure the incremental conversions of a marketing campaign when user-level A/B tests are impossible?
  • How to choose test and control regions for a geo experiment?

Tools and methods

  • Geo experiment — regions, not users, are the experimental units.
  • BSTS counterfactual (CausalImpact R package) — predicts test-region performance without the campaign.
  • MarketMatching / stratified sampling — pick comparable control regions or build representative test samples.
  • A/A test and simulated power analysis — validate the model and size the test before spending budget.
Continue reading — it's free

Google or a one-time email code. No password.