Problem
Marketing teams spend across many channels, and most measurement methods credit an ad for any purchase that follows it. Correlation does not show whether the customer would have bought anyway. A user-level test, with some users shown the ad and some held out, is impossible for TV and billboards. For digital ads it has also become unreliable because of privacy rules (iOS 14, GDPR), multiple devices and private browsing. Without a causal estimate, budget can go to channels that only look good in reports.
Example from the article: Expedia Group runs geo tests across several brands and markets to decide where to put marketing capital.
Solution in brief
Use geographic regions as the experimental units. Show ads in some regions and not in others, build a synthetic control from untreated regions, and measure the difference in a response metric. Validate the design before launch, and compute lift and iROAS afterwards.
Questions this case answers
- How to estimate the causal effect of an intervention when individual units cannot be randomized and only aggregated regional data is available
- How to build a synthetic control from untreated regions and validate it before the intervention starts
- How to measure the incremental effect of advertising when users cannot be split into test and control groups?
- How to run a geo experiment: choose geos, assign treatment and control, and validate the design before spending budget?
Tools and methods
- Geo experiment — treatment and control are regions, so the test needs no user-level data and works for offline and digital media.
- Synthetic control — a weighted mix of control geos that tracks the treated geos before the test.
- Power analysis — predicts the precision of the test and the chance of a significant result before budget is committed.
- iROAS — incremental return on ad spend, the cost-efficiency measure.
Google or a one-time email code. No password.