Path to high-quality LLM-based Dasher support automation

Zhe Jia, Shuai Wang, Aditi Bamba, Martin Wang17 Sep 2024

Problem

DoorDash couriers ("Dashers") contact support when a delivery goes wrong. The old automated support used pre-built resolution flows, so only a small share of issues was solved fast. The knowledge base had the answers, but articles were hard to find, slow to read and only in English. A RAG chatbot fits, but LLM answers can sound right and still be wrong, drift from the retrieved text, ignore the reply language or take too long.

Example from the article: a Dasher describes a problem over several messages. The bot must first work out the real issue, then answer from the right article, in the Dasher's language, without inventing policy.

Solution in brief

A RAG chatbot answers from knowledge base articles found through similar past cases. Every reply passes a two-tier guardrail (cheap semantic similarity first, an LLM evaluator second). Quality is tracked by an LLM judge whose open-ended findings become fixed monitors, calibrated by human review. Prompt changes are gated by regression tests.

Questions this case answers
  • How to ground a conversational assistant in a curated document base and verify each answer before the user sees it
  • How to turn open-ended LLM-judge findings into stable, countable quality monitors for a high-volume assistant
  • How to build a support chatbot that answers from a knowledge base instead of from the model's own memory?
  • How to stop a RAG chatbot from showing hallucinated or policy-violating answers without making every reply slow and expensive?

Tools and methods

  • RAG — answers built from knowledge base articles retrieved for the summarized issue
  • Embedding similarity guardrail — cheap first check of a response against article segments
  • LLM guardrail — second-tier evaluator for groundedness, coherence and compliance
  • LLM judge — scores transcripts on five quality aspects
  • Promptfoo — open-source test runner that blocks failing prompt changes
  • Human review — support agents as experts and calibration of the automated monitors
Continue reading — it's free

Google or a one-time email code. No password.