Problem
Most Yandex employees are programmers and use Yandex Search for professional queries. For a long time they felt that programming search was poor, while the product metrics showed no problem. Labels came from assessors who are qualified but not programmers, so they could not tell a good answer about find_if from a page that merely looks tidy. Models trained on these labels then ranked results for specialists.
Example from the article: employees gathered one evening, labeled real programming queries pairwise over pizza, and the resulting 30 judgments already showed a problem.
Solution in brief
First rebuild the measurement: programmers label programming queries, and their labels are checked against the team's own view. Then improve the stack: a separate transformer for programming, a distilled query classifier to switch it on, and a dedicated ranker later merged into the main one.
Questions this case answers
- How to verify that an offline relevance metric reflects the needs of a specialist user segment when generalist annotators cannot judge the content
- How to adapt a production ranking stack to a small, expert-heavy query segment without raising the number of failure points
- How to check that a search quality metric reflects what expert users need, when the product metric says everything is fine but the team's own experts are unhappy?
- How to improve search for a narrow professional segment without disturbing the main ranking model?
Tools and methods
- Expert relevance labeling — assessors who pass programming tasks label query–document pairs.
- Pairwise side-by-side judgments — a person sees a query and two documents and picks the better one or marks them comparable.
- CS-YATI — a transformer for programming queries, pre-trained on programming texts, trained on clicks of programmer-like users.
- DSSM query classifier — a light model distilled from CS-YATI that answers "about programming or not".
- CatBoost — gradient-boosting ranker over query–document factors.
Google or a one-time email code. No password.
