Problem
On Thumbtack, service professionals have weekly budgets. A visitor-randomized A/B test breaks when a feature changes how those budgets are spent: professionals are reached by visitors in both groups, so the groups affect each other (cross-bucket interference, or spillover). The result can mislead or settle nothing. Fixed baseline and variant markets avoid this but need a long test and careful balancing.
Example from the article: the Marketplace Matching team wanted to test a feature that optimizes the weekly budget spent in a market, such as House Cleaners in Denver.
Solution in brief
Run a market-based test (MBT) where whole markets, not visitors, are the unit, and switch each market between baseline and variant every week. Thumbtack tried two assignment schemes: randomized (a coin flip per market per week) and checkerboard (a fixed starting split that is reversed each week). Compared with fixed-assignment MBTs, switchbacks give shorter and more sensitive tests.
Questions this case answers
- How to measure the effect of a change when treated and control units draw on the same limited resource and unit-level randomization is biased
- How to cut the duration and raise the sensitivity of an experiment that randomizes whole clusters of units
- How to test a feature when treatment and control visitors draw on the same shared budgets?
- How to shorten a market-based test without losing the ability to detect small effects?
Tools and methods
- Market-based test (MBT) — experiment where a market (a cluster of category and zip-code pairs) is the unit, kept in baseline or variant for the test.
- Switchback test — the same markets alternate between baseline and variant across weeks.
- Randomized switchback — each market's assignment is re-drawn at random every week.
- Checkerboard switchback — a pre-balanced split of markets whose assignments are flipped every week.
Google or a one-time email code. No password.