Problem
System analysts at CoronaTech work with interlinked statuses, transfers and loans described in Confluence, where built-in search is poor. The team wanted an assistant that answers from this documentation. Naive RAG (fixed-size cuts, nearest vectors) loses diagrams, screenshots and the context of each fragment. A wrong answer is costly: the asker does not know the right answer and will believe a hallucination.
Example from the article: the first run on a 400-question test set scored 75%, with 25% of answers "zeros" (no answer from the documentation).
Solution in brief
Convert pages to Markdown, turn images into text, and let an LLM cut pages into self-contained chunks with a path and context. Search three vectors per chunk (two dense, one sparse), merge with Reciprocal Rank Fusion (RRF), and filter the top 50 with an LLM. Measure on 400 hand-written question-answer pairs. Quality rose from 75% to 90%.
Questions this case answers
- How to build a question-answering assistant over an internal document base with diagrams and screenshots so that answers are reliable and checked against a human-written test set
- How to split documents into self-contained chunks with their path and context so that retrieval does not return misleading fragments
- How to build a RAG assistant over internal documentation (Confluence) that answers reliably
- How to combine dense and keyword search with reranking in a RAG pipeline
Tools and methods
- LLM-driven chunking — 5-10 sentence units of meaning via Structured Output
- Chunk context — an LLM describes the role of each chunk in its page
- FRIDA embedder — dense vectors for Russian and English
- gte-multilingual-base sparse output — neural keyword search
- Reciprocal Rank Fusion — merges the three result lists
- LLM reranking — a 0/1 relevance filter per chunk
- Open WebUI, Qdrant — interface with ratings; vector store
- LLM-as-judge with human validation — 0/1/2 scale
Google or a one-time email code. No password.