From Zero to Agents: Growing a Multi-Agent LLM Platform Inside a Company

Alexander Konstantinov23 May 2026

Problem

An analytics team at Chestny Znak wanted an assistant over its own Confluence: spec lookup and comparison, SQL drafts, onboarding questions. A bare LLM answers confidently without knowing the pages. Pages can contain passwords, so only open-source models inside the company perimeter were allowed.

The first version put whole articles into the prompt, up to 120K tokens per request. Expert labeling arrived half a year late, so neither retrieval nor answers could be measured.

Example from the article: a user asks for existing specifications that already solve a similar task; the assistant must find and compare the pages.

Solution in brief

Re-chunk the knowledge base along Markdown structure and measure retrieval: context fell about 3 times. Build a synthetic question set with a large model, so generation is scored without user labels. Turn RAG from a fixed pipeline into one tool of a ReAct agent: Overall rose from 0.638 to 0.703.

Questions this case answers
  • How to make a document assistant answer accurately with far less context per request when the whole knowledge base cannot fit into a prompt and the models must run inside a closed perimeter
  • How to evaluate retrieval and generation quality of a document assistant when users rarely give ratings and no labeled answers exist
  • How to make a corporate RAG assistant answer accurately with far less context per request
  • How to evaluate RAG answer quality when users cannot supply labels

Tools and methods

  • vLLM — production inference of open-source models on rented GPUs.
  • Qdrant — vector store with dense and sparse vectors and HNSW.
  • Markdown-aware chunking with Small-to-Big — retrieval on small units, a bigger context around the hit.
  • RAGAS — synthetic question set built from a knowledge graph; Faithfulness, Answer Relevancy and Answer Correctness metrics.
  • ReAct agent (LangGraph) — calls Qdrant search, abbreviation expansion and context reranking itself.
Continue reading — it's free

Google or a one-time email code. No password.