Greeting + Does RAG Even Work?

Ivan Komarov, Evgeny Chukanov7 Nov 2025

Problem

System analysts at CoronaTech work with interlinked statuses, transfers and loans described in Confluence, where built-in search is poor. The team wanted an assistant that answers from this documentation. Naive RAG (fixed-size cuts, nearest vectors) loses diagrams, screenshots and the context of each fragment. A wrong answer is costly: the asker does not know the right answer and will believe a hallucination.

Example from the article: the first run on a 400-question test set scored 75%, with 25% of answers "zeros" (no answer from the documentation).

Solution in brief

Convert pages to Markdown, turn images into text, and let an LLM cut pages into self-contained chunks with a path and context. Search three vectors per chunk (two dense, one sparse), merge with Reciprocal Rank Fusion (RRF), and filter the top 50 with an LLM. Measure on 400 hand-written question-answer pairs. Quality rose from 75% to 90%.

Questions this case answers
  • How to build a question-answering assistant over an internal document base with diagrams and screenshots so that answers are reliable and checked against a human-written test set
  • How to split documents into self-contained chunks with their path and context so that retrieval does not return misleading fragments
  • How to build a RAG assistant over internal documentation (Confluence) that answers reliably
  • How to combine dense and keyword search with reranking in a RAG pipeline

Tools and methods

  • LLM-driven chunking — 5-10 sentence units of meaning via Structured Output
  • Chunk context — an LLM describes the role of each chunk in its page
  • FRIDA embedder — dense vectors for Russian and English
  • gte-multilingual-base sparse output — neural keyword search
  • Reciprocal Rank Fusion — merges the three result lists
  • LLM reranking — a 0/1 relevance filter per chunk
  • Open WebUI, Qdrant — interface with ratings; vector store
  • LLM-as-judge with human validation — 0/1/2 scale
Continue reading — it's free

Google or a one-time email code. No password.