Problem
A recommender that only follows immediate interests misses what the user is trying to achieve across sessions, such as planning a wedding or renovating a kitchen. Pinterest wanted notifications tied to those goals, which it calls user journeys. The team had no large labeled dataset for the new task, and a user can run several overlapping journeys at once. The baseline to beat was interest-based notifications.
Example from the article: a user with the journeys "Kitchen Remodel", "Engagement Ring Selection" and "Christmas DIY Crafts", each with keywords, a stage and a confidence score.
Solution in brief
Treat every query, annotation and interest tag of a user as a keyword, embed the keywords with a pretrained model and cluster them hierarchically. Each cluster is a journey candidate. Small models then rank the candidates, diversify them, name them and mark them ongoing or ended, and the result is stored per user for notifications.
Questions this case answers
- How to group a user's multi-session activity into distinct goal-oriented threads when no labeled taxonomy of goals exists
- How to rank and diversify a user's inferred interest groups so a downstream message does not repeat the same theme
- How to infer a user's long-term goals, not just current interests, from search and engagement history?
- How to build a first journey-detection system when there is almost no training data?
Tools and methods
- Hierarchical clustering on pretrained keyword embeddings — turns a user's keywords into journey candidates.
- Pointwise ranking model — orders a user's journeys by engagement and recency.
- Similarity penalty re-ranking — keeps the top journeys from being near-duplicates.
- Engagement-based stage classification — labels journeys evergreen or situational, ongoing or ended.
- LLM judge — scores journey relevance on a 5-level scale instead of human raters.
- Qwen fine-tuned on GPT labels, Ray batch inference — replaces the multi-model pipeline with one LLM.
Google or a one-time email code. No password.