Leveraging Switchbacks: Enhancing Market-Based Testing at Thumbtack

Dhananjay Sathe, Wade Fuller19 Jul 2023

Problem

On Thumbtack, service professionals have weekly budgets. A visitor-randomized A/B test breaks when a feature changes how those budgets are spent: professionals are reached by visitors in both groups, so the groups affect each other (cross-bucket interference, or spillover). The result can mislead or settle nothing. Fixed baseline and variant markets avoid this but need a long test and careful balancing.

Example from the article: the Marketplace Matching team wanted to test a feature that optimizes the weekly budget spent in a market, such as House Cleaners in Denver.

Solution in brief

Run a market-based test (MBT) where whole markets, not visitors, are the unit, and switch each market between baseline and variant every week. Thumbtack tried two assignment schemes: randomized (a coin flip per market per week) and checkerboard (a fixed starting split that is reversed each week). Compared with fixed-assignment MBTs, switchbacks give shorter and more sensitive tests.

Questions this case answers
  • How to measure the effect of a change when treated and control units draw on the same limited resource and unit-level randomization is biased
  • How to cut the duration and raise the sensitivity of an experiment that randomizes whole clusters of units
  • How to test a feature when treatment and control visitors draw on the same shared budgets?
  • How to shorten a market-based test without losing the ability to detect small effects?

Tools and methods

  • Market-based test (MBT) — experiment where a market (a cluster of category and zip-code pairs) is the unit, kept in baseline or variant for the test.
  • Switchback test — the same markets alternate between baseline and variant across weeks.
  • Randomized switchback — each market's assignment is re-drawn at random every week.
  • Checkerboard switchback — a pre-balanced split of markets whose assignments are flipped every week.
Continue reading — it's free

Google or a one-time email code. No password.