Evaluation of RAG Metrics using RAGA

Harika Samala8 Jul 2024

Problem

A team that ships a retrieval-augmented generation (RAG) assistant needs to know whether bad answers come from the retriever or from the language model. One overall "is the answer good" score does not show which part to fix.

Example from the article: the author scores a question-answering RAG on the FiQA financial dataset and finds mean answer correctness of only 0.48 while answer similarity is 0.88.

Solution in brief

Split evaluation by stage. Retrieval is scored with context recall and context precision, generation with faithfulness and answer relevancy, and the whole pipeline with answer similarity and answer correctness. The RAGAS library computes all six by asking an LLM to break texts into statements and judge them, plus embedding similarity.

Questions this case answers
  • How to localize quality loss in a two-stage retrieve-then-generate pipeline when only end-to-end answers and reference answers are available
  • How to score free-text answers automatically without human raters, using a judge model and embeddings
  • How to measure whether a RAG system's weak answers come from retrieval or from generation?
  • How to score a RAG pipeline automatically against a set of reference answers?

Tools and methods

  • RAGAS — library that computes the six RAG metrics per row of an evaluation table
  • Azure OpenAI (gpt-35-turbo, text-embedding-ada-002) — judge LLM and embeddings in the article's setup
  • LangChain — wrappers that pass the LLM and embeddings to RAGAS
  • FiQA ragas_eval subset — test table with questions, answers, contexts and ground truths
Continue reading — it's free

Google or a one-time email code. No password.