Problem
Netflix runs user acquisition campaigns for its games on Google, Meta and TikTok, mostly at the country level, so everyone in a targeted country sees the ads and no internal control group exists. Because members buy a subscription rather than a single game, the company only sees signups and not why people signed up. It also needs to forecast engagement and the cash spent on content that has no specific title yet. Without a counterfactual, campaign lift, signup attribution and forecasts are guesses.
Example from the article: a campaign runs in some countries; a weighted mix of countries without the campaign stands in for what would have happened in the treated ones.
Solution in brief
Four methods are shown. Synthetic control estimates campaign lift and cross-checks signup estimates. A daily state machine (Markov chain) of account states simulates monthly active accounts. A three-step model with a constrained cubic polynomial forecasts cash spend for placeholder titles.
Questions this case answers
- How to estimate the incremental effect of a campaign that runs in whole regions when there is no untreated control group inside those regions
- How to check a counterfactual estimate of incremental signups when no randomized experiment is available
- How to forecast an engagement aggregate or planned spend before details are known and rank which transition rates drive it most
- How to measure the incremental installs and engagement of a country-level ad campaign when no country is left as a control?
Tools and methods
- Synthetic control — weighted combination of unexposed countries as the counterfactual for treated countries.
- Placebo tests — p-value for the estimated lift.
- Incremental acquisition estimate — counterfactual rate of new members starting to play a game (per the paper).
- State machine / Markov chain — daily transition matrix between account states to simulate and run sensitivity analysis.
- Constrained cubic polynomial fit — shape of cash spend inside each production phase.
Google or a one-time email code. No password.