Thinking Fast & Slow: How Databricks Built High-Speed and Deep Research Agents

Michael Bendersky18 Jun 2026

Problem

Teams that put an LLM agent over enterprise documents need two things that usually get separate stacks. Embedded and customer-facing cases (voice agents, support, copilots) need an answer in seconds at a cost of cents. Expert cases (financial due diligence, clinical evidence review, diagnostics) need multi-step planning, verification and long reports, and tolerate minutes of runtime. Two stacks mean two sets of tools, benchmarks and quality work.

Example from the article: the Databricks Knowledge Assistant in instant mode shows the first token in about 2 seconds and finishes in under 8 seconds; the earlier version without the custom model was slower.

Solution in brief

Build both modes on one foundation: a specification (instructions, index schema, constraints) passed through every retrieval stage, the same search tools, and one benchmark. Instant mode uses one model that generates queries and re-ranks in parallel. Thinking mode uses an RL-trained agent that runs several times in parallel, with an aggregator merging the runs.

Questions this case answers
  • How to run a fast low-cost retrieval answer and a slow deep research agent on one shared search stack
  • How to keep a long-running research agent from looping, forgetting or giving up early
  • How to build a fast, cheap instant search and a slow deep research agent on one shared retrieval stack
  • How to cut time-to-first-token and total latency of a RAG assistant without losing retrieval quality

Tools and methods

  • Instructed Retriever / IR1 — specification-propagating retrieval; one SFT+RL model for query generation and re-ranking.
  • Pivot-based re-ranking — parallel group-wise scoring against reference documents.
  • Agentic data synthesis — synthetic training questions filtered by solver pass rate.
  • CARL and CarlBench — RL-trained deep research agent and the shared benchmark.
Continue reading — it's free

Google or a one-time email code. No password.