Retrieval-Augmented Fill-in-the-Middle: An Architecture for Generating Structured Documents with LLMs

Vladimir Skurikhin24 May 2025

Problem

A team had to generate documents for a customer's corporate knowledge base. The input is a template (up to 5000 tokens, headings plus reference text on what each section should contain), source documents (up to 25,000 tokens) and a user instruction. Production context is capped at 8000 tokens, so template and sources do not fit together. The team had no labeled data, one 40 GB GPU and a weak ~8B LLM for synthetic data.

Example from the article: a real customer template with tables of abbreviations and of terms with definitions plus a "General information" section; the generator filled them with meaningful content.

Solution in brief

Split the template by headings and generate each section separately as a fill-in-the-middle (FIM) task. Each prompt combines the instruction, the section's reference text, BM25-retrieved source chunks and the neighbouring text on both sides. A LoRA-tuned instruct model, trained on synthetic data replaces prompt-only generation.

Questions this case answers
  • How to generate a long structured document section by section so that it matches a template's exact heading structure when template plus sources exceed the context window
  • How to build a fine-tuning dataset for structured document generation when there is no labeled data and only a weak small model for synthetic data
  • How to generate documents that follow a template's exact heading structure with a small LLM
  • How to fit a large template and long source documents into an 8000-token context window

Tools and methods

  • Fill-in-the-Middle (FIM) — generate one section given its left context (prefix), right context (suffix) and reference.
  • BM25 over source chunks — keeps each prompt within 8000 tokens.
  • LoRA fine-tuning of Vikhr-Nemo-12B-Instruct — adapts the model to the FIM prompt format.
  • Synthetic and semi-synthetic datasets — training data built with a weak LLM and filters.
  • vLLM + LangServe — serving with swappable LoRA adapters.
  • BERTScore, chrF++, LLM-as-a-judge — offline metrics.
Continue reading — it's free

Google or a one-time email code. No password.