Building a Transformer-Based Category Recommender at Thumbtack

Andrew Morss11 Sep 2026

Problem

Thumbtack has about 500 job categories, and its lifecycle marketing team picks which ones to show a user in push and email notifications. Home-services users act rarely, so search histories are short. A model trained with plain cross-entropy over the whole catalog learned to pick the most popular categories. That is popularity bias: weak personalization and little diversity.

Example from the article: the initial model (the "old model" in the metric table), trained with naive cross-entropy over all categories, reached a hit rate at 10 (HR@10) of 2.135% on tail categories, while head categories reached 89.472%.

Solution in brief

The model is a transformer over tokens built from the user's search history and context features. A candidate category is scored by cross-attention over those tokens. Training replaces full-catalog cross-entropy with adaptive mixed sampling: score the positive plus in-batch and random negatives, keep only the top-K hardest negatives, and compute the loss over those.

Questions this case answers
  • How to prevent a next-item recommender trained with softmax cross-entropy from collapsing onto the most popular items when user history is sparse
  • How to serve several recommendation use cases with different allowed input signals from a single model
  • How to stop a recommender trained on skewed interaction data from always recommending the most popular items?
  • How to measure popularity bias with per-cohort hit rate and a single balance score?

Tools and methods

  • Multi-token transformer with cross-attention — scores a candidate category against history and context tokens; tokens can be masked.
  • Adaptive mixed negative sampling — pool of in-batch and random negatives, keep the top-K highest-scoring ones for the loss.
  • Cohort HR@10 and balance score — hit rate for head, mid and tail categories, plus one summary number.
  • PyTorch — the sampling step.
Continue reading — it's free

Google or a one-time email code. No password.