Effective document management and retrieval for generative AI

Xintia Gyenge, Yevgeniy Ilyin18 Jun 2026

Problem

Teams building RAG over contracts, loan files and other long documents get wrong answers even with a strong LLM. Most of them tune models, embeddings and chunk sizes (about 80% of the customers the speakers meet). The speakers say the failures sit upstream: parsing loses headings and tables, charts and scans are ignored, and retrieval quality is never measured. Nobody can tell which stage broke or whether a change helped.

Example from the article: A bank customer reported that the pipeline returned the wrong loan term. The speakers also cite frontier agents scoring below 50% on the OfficeQA document-reasoning benchmark.

Solution in brief

Treat the pipeline as five stages (source preparation, parsing, extraction and classification, multimodal handling, indexing) and keep document structure through all of them. Retrieve with hybrid search (BM25 plus dense vectors, fused by RRF), add a reranker only if it measurably helps, and evaluate both at index time and in production.

Questions this case answers
  • How to build a retrieval pipeline over long structured documents that is measured stage by stage and keeps headings and tables intact
  • How to decide whether hybrid search or a reranker improves retrieval on a specific document corpus
  • How to build a document retrieval pipeline for RAG that is measured and improved end to end
  • How to find out whether hybrid search or a reranker helps on my document corpus

Tools and methods

  • Hybrid retrieval (BM25 + dense, RRF) — default query mode for document corpora.
  • Cross-encoder reranker — optional rescoring of top-K results.
  • Heading-preserving chunks — heading path and metadata inside the chunk text.
  • Databricks AI Functions and Vector Search — parsing, extraction, chunking, managed index.
  • Retrieval Quality Auto-Eval, MLflow GenAI scorers — index-time and production evaluation.
Continue reading — it's free

Google or a one-time email code. No password.