Problem
Thumbtack has about 500 job categories, and its lifecycle marketing team picks which ones to show a user in push and email notifications. Home-services users act rarely, so search histories are short. A model trained with plain cross-entropy over the whole catalog learned to pick the most popular categories. That is popularity bias: weak personalization and little diversity.
Example from the article: the initial model (the "old model" in the metric table), trained with naive cross-entropy over all categories, reached a hit rate at 10 (HR@10) of 2.135% on tail categories, while head categories reached 89.472%.
Solution in brief
The model is a transformer over tokens built from the user's search history and context features. A candidate category is scored by cross-attention over those tokens. Training replaces full-catalog cross-entropy with adaptive mixed sampling: score the positive plus in-batch and random negatives, keep only the top-K hardest negatives, and compute the loss over those.
Questions this case answers
- How to prevent a next-item recommender trained with softmax cross-entropy from collapsing onto the most popular items when user history is sparse
- How to serve several recommendation use cases with different allowed input signals from a single model
- How to stop a recommender trained on skewed interaction data from always recommending the most popular items?
- How to measure popularity bias with per-cohort hit rate and a single balance score?
Tools and methods
- Multi-token transformer with cross-attention — scores a candidate category against history and context tokens; tokens can be masked.
- Adaptive mixed negative sampling — pool of in-batch and random negatives, keep the top-K highest-scoring ones for the loss.
- Cohort HR@10 and balance score — hit rate for head, mid and tail categories, plus one summary number.
- PyTorch — the sampling step.
Google or a one-time email code. No password.