Problem
Thumbtack ranks local professionals for customer searches in about 500 service categories. In 2020 its ranker was an ensemble of logistic regression models on fewer than 10 handcrafted features, baked into a Go service. Offline gains from new features stopped turning into A/B wins. Iterations were slow, and a new feature needed at least a month of logged data before training. The team also had no estimate of position bias.
Example from the article: the first non-linear model shipped (gradient boosted trees with new features, early 2021) lifted global conversion by 1.4% relative to baseline.
Solution in brief
Thumbtack first made experiments cheap: a feature store with backfill, a separate model inference service, one-day launches and a self-serve analysis dashboard. Then it moved from logistic regression to boosted trees, XGBoost with position-bias correction, and DCN V2, accepting that only 1 in 3 or 4 experiments wins. Over two years 6 of 20+ experiments shipped, for a cumulative +4.5% conversion.
Questions this case answers
- How to turn offline ranking-metric improvements into online wins when experiment iterations are slow and the model class is limited
- How to estimate and remove position bias from click-like feedback when full randomization would hurt users
- How to restart a stalled ranking roadmap when offline metric gains no longer turn into A/B test wins
- How to measure position bias in a marketplace search without fully randomizing the results
Tools and methods
- Feature store with backfill — features from already logged events
- SageMaker inference service — serves non-linear models outside the Go service
- MLflow — experiment tracking
- XGBoost with monotonic constraints — fast inference, product-consistent behavior
- PDP and SHAP — spot features that contradict product logic
- Boruta SHAP — feature selection over 200+ features
- RandPair swaps and IPS — measure and remove position bias
- DCN V2 — neural ranker in the ensemble
Google or a one-time email code. No password.