Problem
Sber's AI team built a RAG chatbot that explains the terms of a customer's personal insurance policy. Each user needs facts from their own contract, so retrieval is unavoidable. The team had to measure answer quality automatically. Overlap metrics (BLEU, ROUGE) against expert answers were available, but they only measure the distance to one expert wording. The open LLM-based evaluators (RAGAS, ARES) were built for English: prompts, sentence splitters and models do not carry over to Russian as is.
Example from the article: a user asks how to insure a bathhouse. The team wants to know whether the answer is understandable, stays within the contract text and invents nothing.
Solution in brief
Compare two open frameworks on the team's own data. RAGAS scores through hard-coded LLM prompts. ARES trains small classifiers on LLM-generated synthetic data. Check each against human labels on 700 and 500 generated examples. Neither wins on every metric.
Questions this case answers
- How to evaluate a retrieval-augmented answer system automatically when there are no reference answers, using LLM-based judges
- How to adapt LLM-based automatic evaluators built for English to another language and check them against human labels
- How to evaluate the quality of a RAG assistant automatically when there are no gold answers
- How to adapt RAGAS or ARES to the Russian language
Tools and methods
- RAGAS — LLM-judge metrics (faithfulness, answer correctness, context relevancy) and synthetic question generation
- ARES — synthetic data plus three fine-tuned binary classifiers, with confidence intervals
- GigaChat, Command R — generators of synthetic questions and answers; GigaChat also acts as judge
- BLEU, ROUGE — baseline similarity to expert answers
- Human annotation — reference for judging the automatic evaluators
Google or a one-time email code. No password.