Problem
A growth team that spends heavily on paid media needs to know what each channel adds beyond what would have happened anyway. Last-touch and multi-touch attribution under-credit view-based, top-of-funnel channels such as video, and over-credit search: a user may see YouTube ads, later search for the brand and convert, and search gets the credit. The end of third-party cookies makes tracking worse. Budget then goes to the wrong channels.
Example from the article: Grammarly ran a YouTube go-dark experiment and measured the incremental effect on new active users.
Solution in brief
Turn the channel off in a set of geographic markets (the test set) and keep it on in the rest (the control set). Pick the two sets so a time-series model trained on history predicts the test KPI accurately. After the intervention, compare actual test performance with a synthetic baseline and read the difference as incremental lift. Size and length come from a simulation-based power analysis.
Questions this case answers
- How to estimate the causal effect of a marketing channel on acquisition when user-level tracking is unavailable and only region-level randomization is possible
- How to choose balanced region groups and the experiment size and length when the number of units is small and switching a channel off has a business cost
- How to measure the incremental effect of a top-of-funnel ad channel when click-based attribution credits the wrong channel?
- How to choose a balanced test/control split of geographic markets for a go-dark experiment?
Tools and methods
- Geo go-dark experiment — regions as experiment units, spend off in test regions.
- Time-based regression (TBR) with Bayesian structural time series — synthetic baseline and daily intervals.
- Split generation: clustering and randomization — candidate test/control splits.
- Power analysis by simulation — choose share of markets and duration.
- CausalImpact — considered, not used: better suited to observational data.
Google or a one-time email code. No password.