Problem
A marketing team buys search ads in an app marketplace and wants to know how many app downloads the ads add. The publisher only lets advertisers compare ad creatives. It has no randomized holdout, geo targeting is unreliable because users can turn off location-based ads, and users can block the advertising identifier (IDFA), so user-level groups cannot be formed. Synthetic control also fails, because campaigns are re-optimized often and the Android versus iOS split keeps changing.
Example from the article: DoorDash could not tell whether its app-marketplace search campaigns drove incremental downloads and new customers.
Solution in brief
Randomize time instead of users. Each day is turned on or off at random, the sequence is mirrored in the next week, and the difference in daily downloads between on and off days is the incremental effect. A bootstrap on historical days, built with the same on/off pattern, gives the 95% interval of normal variation to compare against. A stable download-to-first-order rate converts downloads into new customers.
Questions this case answers
- How to estimate the causal effect of an intervention when user-level randomization, geo targeting and synthetic controls are all unavailable and only the time of exposure can be controlled
- How to judge whether a difference between alternating on/off time periods exceeds normal variation when only a short history is available
- How to measure the incremental app downloads from search ads when the ad platform offers no A/B test, geo-targeting or user-level randomization?
- How to set a noise baseline so that a difference between ads-on and ads-off periods can be called real?
Tools and methods
- Switchback test on days — alternate campaign on and off by calendar day, so each day is a randomized unit
- Bootstrap baseline — resample historical days with the test's pattern to get the 95% interval of the difference without ads
- Conversion-rate scalar — multiply incremental downloads by the stable download-to-first-order rate
Google or a one-time email code. No password.