RAG Evaluation. Evaluate yourself. How to teach LLM to evaluate LLM work

Vladislav Shufinsky25 May 2024

Problem

Magnit Tech builds a RAG chatbot for internal users over company documents. A RAG system has two parts, retrieval and generation, so when answers get worse it is unclear which one is to blame. Human experts are slow and expensive, and nobody can read 100 production requests every hour. Without an automatic score, no pipeline change (a new embedding model, a spell checker) can be judged.

Example from the article: the team wanted to know whether the chatbot needs a spell checker for typos in user questions. Counting corrected typos is not enough: nine of ten typos fixed is useless if the bot still answers "I can't say anything".

Solution in brief

Use a second LLM as a judge and score retrieval and generation separately. RAGAS gives ready-made prompts for synthetic questions and metrics. ARES fine-tunes task-specific judges and adds some human labels for confidence intervals. To test a pipeline change, score the same questions before and after.

Questions this case answers
  • How to score the answer quality of a document-grounded assistant without human labels and find out whether retrieval or generation causes the failure
  • How to generate a synthetic test set of questions over a private document collection to compare two versions of a pipeline
  • How to evaluate the answer quality of a RAG chatbot without human labels
  • How to find out which RAG component (retrieval or generation) loses quality

Tools and methods

  • LLM as judge — a second LLM rates question, context and answer on a 1–5 scale and explains the rating.
  • RAGAS — synthetic test set generation and metrics: faithfulness, answer relevancy, context precision, context recall.
  • ARES — judges fine-tuned on synthetic data, ranking of RAG systems with confidence intervals via prediction-powered inference (PPI).
  • Augmentex and SAGE — inject typos into questions and correct them, for the spell checker case.
Continue reading — it's free

Google or a one-time email code. No password.