How We Got RAG Running in Production

Petr Korolev, Mikhail Dremin25 May 2024

Problem

Cloud.ru runs three cloud platforms and dozens of services, and their documentation is so large that finding an answer takes time. Support keeps answering questions the documentation already covers. The team wanted search that returns an answer grounded in the docs, to cut time to an answer, raise user loyalty and reduce support load. Fine-tuning an LLM on the documentation is expensive, and the documentation keeps changing.

Example from the article: a user types "virtualka" (slang for a virtual machine) and the search must find documentation that says "virtual machine".

Solution in brief

Combine OpenSearch full-text results (with a synonym dictionary from technical writers) and vector results from a bi-encoder, then rerank the merged list with a cross-encoder trained on assessor labels. The bi-encoder is adapted to the domain with GPL, which needs no human labels. The top 3 paragraphs go to an LLM. On the authors' validation set NDCG@3 rises from 0.535876 (BM25) to 0.668312.

Questions this case answers
  • How to add a cross-encoder reranker on top of hybrid lexical and vector search over a large technical document base in a non-English language
  • How to adapt a bi-encoder retriever to a narrow domain without human labels, by generating pseudo-queries and using a stronger model as a teacher
  • How to add RAG-style hybrid search with a cross-encoder reranker to product documentation search
  • How to adapt a bi-encoder to a narrow domain without human labels

Tools and methods

  • OpenSearch full-text search — BM25 with synonyms, stop words, stemming and shingles
  • Bi-encoder (multilingual-e5-large) + Milvus — semantic retrieval, domain-adapted with GPL
  • Cross-encoder (XLM-RoBERTa) — reranks the merged list; pre-trained on mMARCO, then SFT on labeled data
  • GPL and MarginMSE loss — synthetic training of the bi-encoder with cross-encoder pseudo-labels
  • Prefect, Triton, Kubernetes — ETL orchestration, model inference, deployment
Continue reading — it's free

Google or a one-time email code. No password.