Building a Battle-Tested Open-Source Agentic RAG

Ivan Komarov, Evgeny Chukanov23 May 2026

Problem

A team builds a RAG over its own documents and hears that some off-the-shelf RAG "works". It cannot check that: there is no standard benchmark for whole RAG systems, synthetic benchmarks do not fit a specific corpus, and other teams' numbers come from their own data. Wrong citations are the main risk, because an answer not traceable to the right page cannot be trusted.

Example from the article: in the Agentic RAG Legal Challenge (350+ teams, 300 legal PDFs, 900 questions) OpenAI's built-in RAG scored Total 0.362, mostly because page citations were wrong (grounding G=0.45).

Solution in brief

The authors built MORAG, an open-source agentic RAG with hybrid search, LLM reranking and a knowledge map the agent reads like a table of contents. They tuned it against a golden set with a three-level eval and raised Total from 0.36 to 0.78 in the warm-up, mainly by fixing grounding. They finished 14th of 350.

Questions this case answers
  • How to measure whether a retrieval-augmented answer system over a private document corpus is correct when no standard benchmark for whole systems exists
  • How to make cited source locations of generated answers point to the right pages, and treat wrong citations as failures even when the answer text is right
  • How to build a RAG over a document corpus whose answers cite the correct source pages
  • How to check that a RAG system is actually good when there is no standard RAG benchmark

Tools and methods

  • MORAG — open-source RAG with Open WebUI, Docker; sources Markdown, Confluence, PDF, Jira
  • Hybrid search with RRF in Qdrant — dense Qwen3 embeddings, sparse GTE, BM25 with Snowball stemming
  • Contextual retrieval and LLM reranking — document context per chunk, LLM filter on results
  • Knowledge map and agent tools — search, get_document, search_glossary
  • Golden set with three-level checks — structural, LLM plausibility, consistency with documents
  • DRAGON — benchmark and question-generation recipe
Continue reading — it's free

Google or a one-time email code. No password.