Problem
Yandex geosearch feeds a thematic block (GeoTB) on the search page: a card with one organization, its photos and reviews. For navigational queries, where the user already knows which place they want, a wrong organization disappoints or confuses them and, in the worst case, takes half the screen with useless information. At the start of the project, 8% of GeoTB impressions on geo-intent queries in Russia and 15% in Kazakhstan were irrelevant. Assessor labeling did not match the team's relevance criteria.
Example from the article: for the query "Hermitage, ticket with Pushkin card" the block showed the Hermitage pavilion in the town of Pushkin instead of the State Hermitage Museum.
Solution in brief
The team wrote down three relevance criteria, turned them into an LLM labeling prompt and P-tuned a large LLM. They distilled it into a 0.5B encoder that filters the first organization of the production ranking. After three A/B tests and serving optimizations, bad impressions fell from 8% to 2.5% in Russia, with +3% CTR.
Questions this case answers
- How to reduce irrelevant single-entity results for navigational queries when assessor labels do not match the team's own relevance criteria, then distill an expensive LLM labeler into a cheap production filter
- How to find where a search pipeline loses the relevant result across indexing, candidate generation, ranking and filters
- How to reduce irrelevant results shown in a block that displays a single entity for a navigational query
- How to define and label relevance when assessor labeling does not fit the product's criteria
Tools and methods
- LLM labeling with P-tuning (YandexGPT Pro) — judges a query–organization pair; replaces assessors
- Distillation chain YandexGPT Pro → BoBERT 0.5B → DSSM — makes labeling and inference cheap
- BoBERT 0.5B on GPU (L40) — the production filter
- Loss funnel by search stage — shows where relevant organizations are lost
- fp8 quantization, shorter prompts, same-DC placement — fit the latency and GPU budget
Google or a one-time email code. No password.
