Problem
Marketing teams measure campaign lift with geo experiments. They treat some regions, hold out others and compare the treated regions against a synthetic control. Each test needs representative test and control regions, enough geographic spread to survive a regional anomaly, and the right region size. Regions used in one test cannot be reused until the effect cools down, which caps the number of tests per year. Expedia Group runs up to 40 geo experiments a year across three brands, so many data scientists must set up tests in one standard way.
Example from the article: Expedia splits a national market into segments once, then rotates segments between Test, Control and business-as-usual (BAU) over a quarter to a year.
Solution in brief
Segment the market before the first test. Cluster regions by correlation and size, split every cluster evenly across n segments, then validate the segments with indexed time series, pairwise t-tests and a geographic-spread check. Rotate whole segments through Test, Control and BAU so that tests can be planned ahead and do not overlap.
Questions this case answers
- How to partition a set of units into balanced, representative groups up front so that many sequential experiments can reuse them without carry-over between tests
- How to verify before launch that experiment groups are equivalent to each other and to the whole population when user-level randomization is impossible
- How to split a market into geographic segments that stay representative of the whole market and can be reused across many geo experiments?
- How to run dozens of geo experiments a year without test and control regions contaminating each other?
Tools and methods
- Geo experiment — treat regions, compare with a synthetic control, read lift
- k-means clustering — group regions into "types" by correlation and size
- Student's t-test — check that segments do not differ on new customers, app users and other dimensions
- Synthetic control — modelled counterfactual from control regions, also used for the dry run
Google or a one-time email code. No password.