A Hallucination Detector in RAG Systems

Viktor Nosko, Zakhar Ponimash, Marat Potanin25 May 2024

Problem

Teams that run a RAG assistant over documents (law, medicine, industry) still get answers with invented or distorted facts. Users trust fluent generated text, and the answer cannot be checked without rereading the sources, so the risk lands on production use in critical domains. Retrieval adds to it: cosine similarity over equal-size chunks misses relevant passages. The team needed a cheap score of how much of an answer the documents support.

Example from the article: an answer is split into sentences, and for each sentence the check returns the index of the supporting document sentence or -1, for example [-1, 7, 7, -1, -1, -1, 0].

Solution in brief

Split the answer and the documents into sentences or snippets, compute a similarity matrix, and keep for each answer sentence the best document sentence above a threshold. The share of answer text with no support is the hallucination estimate, and the same index shows what supports each sentence. Use the score to rank candidate answers or to refuse when too much text is unsupported.

Questions this case answers
  • How to measure how much of a generated answer is supported by source documents by matching answer units to document units
  • How to decide automatically to refuse a generated answer when too much of its text has no support in the sources
  • How to detect hallucinations in RAG answers by checking which answer parts are supported by source snippets
  • How to decide automatically that a RAG system should refuse to answer

Tools and methods

  • Similarity matrix with threshold — maps each answer sentence to its best-matching document sentence
  • SBert embeddings + cosine similarity — the Python variant in ExplainitAll
  • n-gram Jaccard distance — the C# variant in AiFramework
  • Cross-encoder (vectara/hallucination_evaluation_model) — third-party alternative the authors compare against
  • Fractal answer synthesis — the authors' graph-based replacement for chunking, which feeds the same RAG
Continue reading — it's free

Google or a one-time email code. No password.