How a 4B Model Beat GPT at Agentic Retrieval (And How to Train One on a Neon Postgres Corpus)

Ying Hang Seah15 Jun 2026

Problem

A team runs an assistant that answers questions over its own documents and calls a frontier model for every request. Inference is expensive and slow, and most requests do not need that much model. A small fine-tuned model can match it, but fine-tuning needs hard questions with known answers, and a raw corpus has none.

Example from the article: on the GitLab handbook, a trained 4B model scored 0.854 average reward against 0.818 for gpt-5.4, at under $0.001 per request versus $0.01.

Solution in brief

Generate questions from the corpus itself: single-document ones, and multi-hop ones built from pairs of linked documents. The source documents give the ground truth. Train a small open-weight model with reinforcement learning: it searches the corpus with a BM25 tool and is rewarded for citing the right documents and answering correctly.

Questions this case answers
  • How to train a small open model for agentic search over a private document corpus when no labeled questions exist
  • How to generate hard multi-hop questions with known answers from raw documents to use as reinforcement learning rewards
  • How to train a small open-weight model for agentic search over a company document corpus
  • How to cut the inference cost and latency of an internal RAG assistant

Tools and methods

  • Synthetic Q&A generation — an LLM writes single-document and multi-document questions from the corpus, with ground-truth answers.
  • RL fine-tuning of a 4B model — a small model learns to call a search tool; rewards come from the trace and the ground truth.
  • LLM judge — compares the agent's answer with the reference answer for the correctness reward.
  • Neon / Lakebase Postgres with pg_search and pgvector — stores the corpus and provides BM25 and vector retrieval.
  • Castform — the speaker's training platform.
Continue reading — it's free

Google or a one-time email code. No password.