Method
Relevance labeling
Producing relevance judgments with assessors, crowd or LLMs for ranking evaluation.
20 cases from 11 companies apply Relevance labeling. 6 were checked in an online A/B test; 8 report results in production.
Typical tasks: Rank search results, Evaluate models, Label data with LLMs, Retrieve and rank from millions of items and 7 more. Companies: Yandex, DoorDash, Pinterest, Mercari, Criteo, Indeed and 5 more. Often combined with: LLM-as-a-judge, Knowledge distillation, Supervised fine-tuning, Side-by-side evaluation, Two-tower model and 49 more.



