Problem
Teams that build LLM products, RAG systems and agents must judge answers that exact-match metrics cannot score. A correct answer with a synonym or extra filler counts as wrong, and style, safety or absence of hallucinations are hard to formalize. The fallback is reading five to ten examples by eye or hiring slow annotators. The cost: model versions are compared on impressions, and quality drops go unnoticed.
Example from the article: a RAG system is monitored online on a helpfulness criterion, so a degrading model is noticed in time and a new version is released or new documents are loaded.
Solution in brief
Use a trained judge model that gets the task, the answer, a criterion and a detailed scale, and returns a score with a comment. Check it against human scores on a held-out set. SBER built Pollux, a small open judge, and a video version, Pollux Vision.
Questions this case answers
- How to replace manual grading of free-form LLM answers with a calibrated automated judge validated against human ratings
- How to evaluate style, safety or hallucinations in generated text automatically when exact string match fails
- How to replace manual grading of LLM answers with a calibrated LLM judge validated against human ratings
- How to evaluate hard-to-formalize qualities of an answer (style, safety, hallucinations) automatically
Tools and methods
- LLM-as-a-judge — a model scores another model's answer against a criterion and a scale.
- Pollux — open text judges (three models, newest 4B) trained on 58 criteria; runs on one GPU.
- Pointwise and pairwise scoring — one answer gets a score on a scale, or two answers are compared.
- Validation against human scores — correlation, macro F1 and RMSE.
- Hugging Face — where the speaker says the models and benchmark details are.
Google or a one-time email code. No password.