Problem
A team connects an LLM to company documents with LlamaIndex or LangChain, and the demo looks convincing. In production the same question gets different answers on different runs, and some answers are wrong. Even a 10% hallucination rate destroys user trust and blocks the rollout. Questions that carry legal or financial risk are the worst case.
Example from the article: given three 2022 annual reports (Bellevue Group, Christian Dior, UNIQA Insurance Group) and the question "Which company has the most liquidity?", the same unchanged script answered Bellevue Group, then UNIQA. The correct answer is Christian Dior, with 7.588 billion EUR in cash and cash equivalents.
Solution in brief
Do not search chunks across all documents. Route each question class to a small, curated knowledge map (a table, a set of guides, a folder) that mirrors how a domain expert looks for the answer, so the LLM sees only the data it needs. When an answer is wrong, tracing shows whether routing, the map or synthesis failed.
Questions this case answers
- How to stop an LLM assistant over long documents from giving unstable or wrong answers by routing each question to a small curated source instead of searching all chunks
- How to locate whether a wrong LLM answer comes from routing, the retrieved data or the generation step so that domain experts can audit it
- How to stop a RAG assistant over business documents from giving wrong or unstable answers
- How to find out why a RAG system hallucinates: routing, retrieved data or synthesis
Tools and methods
- Knowledge map — preprocessed, expert-readable data (table, guides, folder) for one question class
- Router / expert tree — a first-line prompt picks a specialist assistant, each with its own map and search tools
- Text-to-SQL over a map — for tables too big for the context
- Tracing — view the exact prompt of every LLM call
- GPT-4 class model — for synthesis
Google or a one-time email code. No password.