RAG as a Flexible Method for Creating Language Assistants

Kirill Belyakov25 May 2024

Problem

A team built a chatbot for a federal ministry. A support operator passes in a request, and the bot answers from orders and regulations on state services. The goal was to cut the cost of preparing replies in electronic document management. The data was confidential, so the LLM had to run locally.

A plain RAG baseline scored 54.1% answer correctness. The needed chunk reached the top 4 for only 56.1% of detailed questions, and follow-ups in a chat were often misread.

Example from the article: documents are written in a different language from the one people use to phrase questions, so the right chunk is not found.

Solution in brief

The team split quality into three metrics and tuned each stage on its own: chunks, knowledge base format, chat history. The biggest gains came from merging chunks with a zero-shot NLI check and indexing LLM-generated questions per chunk. Embedder fine-tuning barely helped.

Questions this case answers
  • How to raise answer accuracy of a locally hosted document-grounded assistant by tuning chunking, index format and multi-turn history separately, each measured on its own metric
  • How to make a retriever find the right passage when the document language differs from the way users phrase their questions
  • How to raise the accuracy of a locally hosted RAG assistant over regulatory documents
  • How to prepare and merge document chunks so that retrieval finds the right passage

Tools and methods

  • RAG baseline — 500-token chunks, multilingual-e5-large, FAISS, top 4 chunks, Saiga3-8B.
  • MarkerPDF, MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter — split PDFs along headings.
  • Zero-shot NLI chunk merging — joins neighbouring chunks that belong together.
  • Chunk plus generated questions — LLM writes questions for each chunk; questions are indexed, the chunk is returned.
  • Triplet fine-tuning of the embedder — tried twice, small effect.
  • LLM fine-tuning on synthetic chat data — history understanding, summarization, no repeated answers.
Continue reading — it's free

Google or a one-time email code. No password.