Accelerating Ranking Experimentation at Thumbtack with Interleaving

Derek Zhao, Dhananjay Sathe26 Jul 2023

Problem

Thumbtack's consumers search for professionals through ML rankers that can change through new features, model architectures or training techniques. A typical ranking A/B test takes many weeks. That allows few tests, while ranker improvements are many. The team needed a way to find the better of two rankers in days, not weeks.

Example from the article: a ranker model can be improved through any combination of new features, model architectures and training techniques, and A/B testing alone leaves no time to explore that space.

Solution in brief

Interleaving shows each consumer one merged list built from the control and treatment rankers, and credits a ranker when its professional is contacted. The share of contacts credited to the treatment ranker is tested against 0.5. Thumbtack uses it as stage one to screen many rankers, then runs A/B tests on the best candidates.

Questions this case answers
  • How to pick the better of two ranking models with far less traffic and time than a split-group experiment needs
  • How to validate a faster experiment design against the slower gold-standard test before adopting it
  • How to compare many candidate rankers faster than a multi-week A/B test allows?
  • How to test whether interleaving agrees with A/B results on our own users before relying on it?

Tools and methods

  • Interleaving — merges two rankers' lists into one shown list and credits contacts to the source ranker.
  • One-sample Z-test of proportions — tests whether the preference signal differs from 0.5.
  • A/A interleaving test — checks that the framework and analysis add no bias.
  • Bootstrap analysis — simulates experiments at different sample sizes to measure sensitivity gain.
  • A/B test — confirms finalists and measures company KPIs.
Continue reading — it's free

Google or a one-time email code. No password.