Problem
DoorDash couriers ("Dashers") contact support when a delivery goes wrong. The old automated support used pre-built resolution flows, so only a small share of issues was solved fast. The knowledge base had the answers, but articles were hard to find, slow to read and only in English. A RAG chatbot fits, but LLM answers can sound right and still be wrong, drift from the retrieved text, ignore the reply language or take too long.
Example from the article: a Dasher describes a problem over several messages. The bot must first work out the real issue, then answer from the right article, in the Dasher's language, without inventing policy.
Solution in brief
A RAG chatbot answers from knowledge base articles found through similar past cases. Every reply passes a two-tier guardrail (cheap semantic similarity first, an LLM evaluator second). Quality is tracked by an LLM judge whose open-ended findings become fixed monitors, calibrated by human review. Prompt changes are gated by regression tests.
Questions this case answers
- How to ground a conversational assistant in a curated document base and verify each answer before the user sees it
- How to turn open-ended LLM-judge findings into stable, countable quality monitors for a high-volume assistant
- How to build a support chatbot that answers from a knowledge base instead of from the model's own memory?
- How to stop a RAG chatbot from showing hallucinated or policy-violating answers without making every reply slow and expensive?
Tools and methods
- RAG — answers built from knowledge base articles retrieved for the summarized issue
- Embedding similarity guardrail — cheap first check of a response against article segments
- LLM guardrail — second-tier evaluator for groundedness, coherence and compliance
- LLM judge — scores transcripts on five quality aspects
- Promptfoo — open-source test runner that blocks failing prompt changes
- Human review — support agents as experts and calibration of the automated monitors
Google or a one-time email code. No password.