How Etsy Uses LLMs to Improve Search Relevance

Yuqing Zhang, Congzhe Su, Susan Liu16 Jan 2026

Problem

Etsy's search models learned relevance from clicks, add-to-carts and purchases. These signals favor popular listings, even when a less popular listing matches the query better. Human labels fix this but are too slow and expensive for every query-listing pair, and an off-the-shelf LLM does not know Etsy's vocabulary and is too costly to run at scale. The team needed a relevance measure that is separate from engagement and cheap enough to compute daily and inside live search.

Example from the article: a search for "fall decor" returned loosely related items such as clothing before the change, and mostly seasonal decor after it.

Solution in brief

Humans label a golden set. A strong LLM is aligned with it, then labels millions of pairs. Those labels train a smaller fine-tuned LLM for daily offline measurement, which is distilled into a BERT-based two-tower model that filters and scores results in real time.

Questions this case answers
  • How to measure and improve the match between a query and a result when engagement signals are biased toward popular items
  • How to turn an expensive LLM judge into a model cheap and fast enough for daily monitoring and real-time serving
  • How to measure search relevance at scale when clicks and purchases are biased toward popular listings?
  • How to get a cheap LLM relevance judge that agrees with human annotators?

Tools and methods

  • o3 with few-shot chain-of-thought prompting — the LLM annotator that labels pairs as relevant, partially relevant or irrelevant
  • LangGraph — implements the annotator prompt
  • Qwen 3 VL 4B — the teacher, fine-tuned on the annotator's labels
  • BERT-based two-tower model — the student for real-time inference
  • vLLM — high-throughput batch inference for the teacher
  • Golden set of human labels — the reference every model is checked against
Continue reading — it's free

Google or a one-time email code. No password.