A Simulation and Evaluation Flywheel to Develop LLM Chatbots

Lewis Warne, Chenran Gong, Aditi Bamba, Matt Gode26 Jan 2026

Problem

DoorDash Support replaced hand-built decision trees with LLM chatbots for customers and Dashers (couriers). A tree change had a traceable effect. An LLM does not: after a prompt edit the bot may handle one scenario better and another worse, and nobody can predict which. The two usual ways to check are both bad. Shipping to production risks the customer experience, and manual testing of many scenarios is slow and misses cases.

Example from the article: human reviewers saw the chatbot drown in raw tool events and logs in its context window, misread fields and cite policies that do not exist.

Solution in brief

Build an offline simulator in which an LLM plays the customer against the real chatbot with mocked backend tools. Add binary LLM-as-judge evaluations calibrated against expert labels. Then loop: baseline the pass rate on a fixed scenario set, change the prompt or context, re-simulate, re-evaluate, and stop when the pass rate meets the exit criterion.

Questions this case answers
  • How to measure the effect of a prompt or context change on a non-deterministic conversational agent offline, before any live traffic sees it
  • How to calibrate an automated judge against expert labels so that its pass rate can serve as a release criterion
  • How to test a change to an LLM support chatbot without shipping it to customers or reading dozens of conversations by hand?
  • How to build an LLM judge whose verdicts can be trusted as an exit criterion for prompt changes?

Tools and methods

  • LLM customer simulator — plays the customer in multi-turn conversations built from real transcripts
  • Binary LLM-as-judge evaluations — pass/fail verdict with reasoning for one narrow failure mode
  • Calibration against human labels — precision, recall and F1 against expert pass/fail labels
  • Mock tools (gRPC, MCP) — controlled backend responses, arrange-act-assert style
  • Amazon S3 — stores generated test scenarios indexed by test ID
Continue reading — it's free

Google or a one-time email code. No password.