Improving Pinterest Search Relevance Using Large Language Models

Han Wang, Mukuntha Narayanan, Onur Gungor, Jinfeng Rao4 Apr 2025

Problem

Pinterest Search must rank Pins (image-based items) by how well they match a query, not only by past engagement. Human-labeled relevance data is small and mostly covers US queries. The best language models read query and Pin text together, but they are too slow and costly to run on every candidate in real time. Without labels the model stays weak in other languages and on seasonal concepts.

Example from the article: the human-labeled set is about 0.3M examples (per the paper), while the daily search logs hold billions of rows from many countries.

Solution in brief

Fine-tune a large language model as a cross-encoder "teacher" on human labels, with rich text built for each Pin. Let the teacher label a huge log of query-Pin pairs, then train a small feed-forward "student" on those labels and serve only the student. The multilingual teacher carries relevance judgments to languages and countries that have no human labels.

Questions this case answers
  • How to score the relevance of every query-item pair at serving scale when only a small human-labeled sample exists and the best model is too slow to serve
  • How to extend a relevance model to languages and markets that have no human labels
  • How to get relevance labels for billions of query-item pairs when human annotation covers only a small, mostly US-based sample?
  • How to use a large language model for ranking relevance when it is too slow and expensive to serve online?

Tools and methods

  • Cross-encoder teacher (best compared: Llama-3-8B with QLoRA) — predicts a 5-level relevance label from query plus Pin text
  • Pin text representation — title, caption, board titles, linked page and top queries as teacher input
  • Knowledge distillation — teacher labels train a lightweight student on billions of logged pairs
  • Student feed-forward network — serves online from embeddings, BM25 and engagement features
Continue reading — it's free

Google or a one-time email code. No password.