Make Retrieval Smarter: How We Built Smart Search over a Document Base

Dmitry Bakanov24 May 2025

Problem

Support specialists at a customer of Innotech spent time on typical requests that the corporate knowledge base already answered, because its keyword search worked poorly. The team had to build a search that finds the right documents and writes an answer from them. It had to index a new article within 15 minutes, answer within 10 seconds and fit on one GPU. The base held more than 5,000 articles plus Office and PDF attachments.

Example from the article: a user asks for the structure requirements of the "Administrator Guide" artifact and gets a point-by-point answer.

Solution in brief

A RAG service with a domain-tuned embedder, hybrid search (vector plus BM25 over title and content, merged by RRF), small chunks expanded with their neighbors, a cross-encoder reranker and an LLM for the answer. Offline, on questions written by the customer's analysts, MRR@5 rose from 0.22 (zero-shot baseline) to 0.802.

Questions this case answers
  • How to answer questions over a large internal document base with a search that merges keyword and vector rankings whose scores are not comparable and indexes new documents quickly
  • How to adapt a text embedding model to a narrow vocabulary without labeled query-document pairs
  • How to replace a weak keyword search over a corporate knowledge base with RAG search that also generates an answer
  • How to adapt an embedding model to a narrow domain without labeled queries

Tools and methods

  • TSDAE, then GPL — two-stage unsupervised tuning of the embedder to the customer's domain.
  • Hybrid search with RRF — vector search and BM25 over title and content, merged by rank.
  • Small2Big chunking — search over small chunks, pass the chunk with its neighbors to the LLM.
  • Cross-encoder or MMR — switchable reranking.
  • MRR@5, chrF++, LLM judges — retrieval and answer metrics.
  • OpenSearch, LangChain, Docling — storage and parsing.
Continue reading — it's free

Google or a one-time email code. No password.