Inside DoorDash’s one-click simulation and evaluation platform for support chatbots

Chenran Gong, Aditi Bamba1 Jun 2026

Problem

A support chatbot is more than a model: prompts, retrieval, knowledge bases, policies and orchestration all shape the answer, and a small change in one layer can shift outcomes elsewhere. DoorDash validated changes on about 1% of users and read transcripts by hand. Review cycles took weeks to months, rare cases (policy edge cases, rare escalation paths) were hard to cover, and silent regressions had no end-to-end baseline.

Example from the article: a customer reports missing items ("3 cookies and the pulled pork"), the in-app report is stuck "under review", and the store does not answer.

Solution in brief

Turn production transcripts into structured scenarios, replay each one with an LLM that plays the customer, and answer the chatbot's tool calls from a mock layer. Then score every transcript with feature-specific LLM-as-a-judge rubrics that return pass or fail. A validation on one feature showed simulated escalation rate close to the live 1% rollout.

Questions this case answers
  • How to validate a tool-using conversational agent before release when live-traffic testing is slow and long-tail scenarios are rare
  • How to build a regression suite for a multi-turn LLM agent from historical conversations while keeping external dependencies deterministic
  • How to test a support chatbot end to end before launch instead of reviewing transcripts from a 1% rollout?
  • How to turn production support transcripts into replayable test scenarios?

Tools and methods

  • Scenario generation from transcripts — an LLM extracts the customer story, behavior traits, issue types and escalation signals; one transcript gives one scenario.
  • LLM customer simulator — plays the customer over several turns, with an optional scripted mode.
  • Mock server and mock-aware routing — returns controlled tool and service responses for simulation traffic only.
  • LLM-as-a-judge rubrics — binary pass/fail checks defined per chatbot feature.
Continue reading — it's free

Google or a one-time email code. No password.