Problem
Teams asked to add an LLM feature often follow the classic ML cycle: collect data, train, measure, repeat. Two failures are common. Nobody needed the task, so half a year yields no business profit. Or current technology cannot solve it, and quality stays below what the business expects. A working prototype must then be evaluated without a ground truth and made cheap enough for production.
Example from the article: DoorDash's support assistant runs a request through RAG and an LLM guardrail, and sends answers judged not good to a human agent.
Solution in brief
Walk through five stops. Pick the model size by task breadth. Prototype with prompts on the strongest model within 2-3 weeks. Split quality into sub-qualities and train a reward model for each. Prefer a graph of LLM components over an agent unless three conditions hold. Cut cost by chained distillation and inference optimizations.
Questions this case answers
- How to decide whether a feature needs a large language model at all and how large the model must be for a task of a given breadth
- How to validate an LLM idea quickly without ground truth and then cut the inference cost of the working prototype
- How to decide whether a product feature needs an LLM and how large the model should be
- How to validate an LLM idea in weeks before collecting data or training anything
Tools and methods
- Prompt Driven Development — a three-step replacement for CRISP-DM that tests the business effect before any data collection
- Sub-quality reward models — separate LLM classifiers for relevance, truthfulness, readability, safety
- Guardrails with human-in-the-loop — a reward score decides whether an answer is shown or handed to a person
- Chained distillation — 30B LLM → 3B LLM → 300M BERT → CNN/FC
- llama.cpp, vLLM, Hugging Face Text Generation Inference — frameworks implementing most inference optimizations
Google or a one-time email code. No password.
