5 Stops on the Way to LLM in Your Production

Vsevolod Vikulin24 May 2025

Problem

Teams asked to add an LLM feature often follow the classic ML cycle: collect data, train, measure, repeat. Two failures are common. Nobody needed the task, so half a year yields no business profit. Or current technology cannot solve it, and quality stays below what the business expects. A working prototype must then be evaluated without a ground truth and made cheap enough for production.

Example from the article: DoorDash's support assistant runs a request through RAG and an LLM guardrail, and sends answers judged not good to a human agent.

Solution in brief

Walk through five stops. Pick the model size by task breadth. Prototype with prompts on the strongest model within 2-3 weeks. Split quality into sub-qualities and train a reward model for each. Prefer a graph of LLM components over an agent unless three conditions hold. Cut cost by chained distillation and inference optimizations.

Questions this case answers
  • How to decide whether a feature needs a large language model at all and how large the model must be for a task of a given breadth
  • How to validate an LLM idea quickly without ground truth and then cut the inference cost of the working prototype
  • How to decide whether a product feature needs an LLM and how large the model should be
  • How to validate an LLM idea in weeks before collecting data or training anything

Tools and methods

  • Prompt Driven Development — a three-step replacement for CRISP-DM that tests the business effect before any data collection
  • Sub-quality reward models — separate LLM classifiers for relevance, truthfulness, readability, safety
  • Guardrails with human-in-the-loop — a reward score decides whether an answer is shown or handed to a person
  • Chained distillation — 30B LLM → 3B LLM → 300M BERT → CNN/FC
  • llama.cpp, vLLM, Hugging Face Text Generation Inference — frameworks implementing most inference optimizations
Continue reading — it's free

Google or a one-time email code. No password.