Problem
Allegro's search team needs to know how relevant the returned products are, not only whether A/B tests move business metrics. Human annotation answers it but is slow and costly: about 30 experts labeled query-product pairs in four languages across 13 departments. Hosted LLM APIs cut the labor, but running them on every daily check is expensive too. The team wanted a judge that agrees with the human labels, runs continuously and is cheap per call.
Example from the article: a compatible third-party phone case must be labeled "Exact product", while a third-party printer toner is only "Highly substitutable".
Solution in brief
Write the judge prompt from the human annotation guidelines, with critical rules and step-by-step instructions and no few-shot examples. Score every product on a listing with it, then compute position-aware metrics such as NDCG. Validate against the 380K+ human-labeled set, and move from a cloud model to a local quantized model where agreement allows (Polish).
Questions this case answers
- How to replace expensive human relevance labeling with an LLM judge that agrees with expert labels, and verify that agreement per class and per language
- How to run a labeling LLM on a large volume of items cheaply when a hosted model is too costly and small models break the output format
- How to measure search relevance continuously without paying for human annotation of every query?
- How to build an LLM judge that agrees with human relevance labels?
Tools and methods
- LLM-as-a-judge — labels each query-product pair on an ordinal relevance scale
- Gemini — cloud judge (Gemini 3.1 Flash Lite as the primary cloud judge, Gemini 2.5 Pro as the quality ceiling)
- Gemma — local judge (26B, 4-bit AWQ, no-thinking)
- Cohen's kappa, quadratic weighted — agreement with human labels
- GEPA — optional prompt refinement on top of an expert prompt
Google or a one-time email code. No password.