Problem
A team had to generate documents for a customer's corporate knowledge base. The input is a template (up to 5000 tokens, headings plus reference text on what each section should contain), source documents (up to 25,000 tokens) and a user instruction. Production context is capped at 8000 tokens, so template and sources do not fit together. The team had no labeled data, one 40 GB GPU and a weak ~8B LLM for synthetic data.
Example from the article: a real customer template with tables of abbreviations and of terms with definitions plus a "General information" section; the generator filled them with meaningful content.
Solution in brief
Split the template by headings and generate each section separately as a fill-in-the-middle (FIM) task. Each prompt combines the instruction, the section's reference text, BM25-retrieved source chunks and the neighbouring text on both sides. A LoRA-tuned instruct model, trained on synthetic data replaces prompt-only generation.
Questions this case answers
- How to generate a long structured document section by section so that it matches a template's exact heading structure when template plus sources exceed the context window
- How to build a fine-tuning dataset for structured document generation when there is no labeled data and only a weak small model for synthetic data
- How to generate documents that follow a template's exact heading structure with a small LLM
- How to fit a large template and long source documents into an 8000-token context window
Tools and methods
- Fill-in-the-Middle (FIM) — generate one section given its left context (prefix), right context (suffix) and reference.
- BM25 over source chunks — keeps each prompt within 8000 tokens.
- LoRA fine-tuning of Vikhr-Nemo-12B-Instruct — adapts the model to the FIM prompt format.
- Synthetic and semi-synthetic datasets — training data built with a weak LLM and filters.
- vLLM + LangServe — serving with swappable LoRA adapters.
- BERTScore, chrF++, LLM-as-a-judge — offline metrics.
Google or a one-time email code. No password.