Method

Relevance labeling

Producing relevance judgments with assessors, crowd or LLMs for ranking evaluation.

20 cases from 11 companies apply Relevance labeling. 6 were checked in an online A/B test; 8 report results in production.

Typical tasks: Rank search results, Evaluate models, Label data with LLMs, Retrieve and rank from millions of items and 7 more. Companies: Yandex, DoorDash, Pinterest, Mercari, Criteo, Indeed and 5 more. Often combined with: LLM-as-a-judge, Knowledge distillation, Supervised fine-tuning, Side-by-side evaluation, Two-tower model and 49 more.

Automating Search Relevance Assessment at Scale with LLM-as-a-Judgee-commerce2026-08
Introducing CLEPR, our model for semantic understandingadtech2026-06
AI Blender in Search: How We Stopped Worshipping Clicks and Moved Block Placement to LLMssearch-engine2026-05
LLM-as-a-Judge: Evaluating natural language searchfood-delivery2026-05
LLM-Powered Relevance Assessment for Pinterest Searchsocial2025-12
Search Results Quality Monitoring with LLMsmarketplace2025-12
Normalized Entropy or Apply Rate? Evaluation Metrics for Online Modeling Experimentsmarketplace2025-11
Optimizing Relevance When Sorting by Price in Marketmarketplace2025-05
Cold-Starting Sales of Complementary Products with LLMsfood-delivery2025-05
Developing Multimodal Content Models for Recommendationssocial2025-05
YandexGPT as a RAG assistant for tech support: solving problems fastergeneral2025-05
LLM-Based Personalization of Alice's Conversations with Usersgeneral2025-05
How DoorDash leverages LLMs to evaluate search result pagesfood-delivery2025-04
Improving Pinterest Search Relevance Using Large Language Modelssocial2025-04
Unleashing the power of large language models at DoorDash for a seamless shopping adventurefood-delivery2024-10
Profit Prophet: Fortune-Telling on Cushions, or How to Evaluate the Effectiveness of a New Search Systeme-commerce2024-05
The Journey to Machine-Learned Re-rankingmarketplace2023-01
How programmers reimagined search for themselvessearch-engine2022-06
Building a Relevance Model for Text and Video Informationmedia-streaming2022-06
Reviews in Organization Search on Yandex Mapssearch-engine2022-06