Problem
A team runs an assistant that answers questions over its own documents and calls a frontier model for every request. Inference is expensive and slow, and most requests do not need that much model. A small fine-tuned model can match it, but fine-tuning needs hard questions with known answers, and a raw corpus has none.
Example from the article: on the GitLab handbook, a trained 4B model scored 0.854 average reward against 0.818 for gpt-5.4, at under $0.001 per request versus $0.01.
Solution in brief
Generate questions from the corpus itself: single-document ones, and multi-hop ones built from pairs of linked documents. The source documents give the ground truth. Train a small open-weight model with reinforcement learning: it searches the corpus with a BM25 tool and is rewarded for citing the right documents and answering correctly.
Questions this case answers
- How to train a small open model for agentic search over a private document corpus when no labeled questions exist
- How to generate hard multi-hop questions with known answers from raw documents to use as reinforcement learning rewards
- How to train a small open-weight model for agentic search over a company document corpus
- How to cut the inference cost and latency of an internal RAG assistant
Tools and methods
- Synthetic Q&A generation — an LLM writes single-document and multi-document questions from the corpus, with ground-truth answers.
- RL fine-tuning of a 4B model — a small model learns to call a search tool; rewards come from the trace and the ground truth.
- LLM judge — compares the agent's answer with the reference answer for the correctness reward.
- Neon / Lakebase Postgres with
pg_searchandpgvector— stores the corpus and provides BM25 and vector retrieval. - Castform — the speaker's training platform.
Google or a one-time email code. No password.