Building Ask DoorDash (Part 3): Evaluation

Steven Xu, Fabio Flores30 Jun 2026

Problem

A team launching a conversational shopping agent sees quality only through employee feedback and manual tests. Those signals are sparse and skewed toward known scenarios. The agent acts through hidden tool calls, so the visible reply does not show whether the user was helped. A change cannot be tried on real users, and a past session cannot be replayed against a modified agent.

Example from the article: Ask DoorDash, a multi-turn restaurant discovery and grocery agent, went from an average of 1 employee-submitted feedback to 2,000 auto-graded sessions per day.

Solution in brief

Turn quality into binary rubric criteria. Rebuild each traced session into a compact view per criterion, and let an LLM judge calibrated on human labels grade it. Use the same rubric and judge on live sessions and on offline sessions from a simulated user working against fixed tool data. Cluster failures by violated criterion.

Questions this case answers
  • How to estimate the quality of an open-ended conversational agent when a goal has many valid outcomes and the hidden tool calls matter as much as the visible reply
  • How to validate a change to a non-deterministic agent offline, with repeatable inputs, before it reaches users
  • How to measure the quality of a multi-turn tool-calling agent at scale instead of relying on employee feedback?
  • How to build an LLM judge that agrees with human reviewers and grades only the evidence a criterion needs?

Tools and methods

  • Rubric with binary criteria — defines a good session; criteria roll up into a session score
  • Transcript builder — turns OpenTelemetry traces in ClickHouse into criterion-specific views
  • LLM-as-a-judge — grades each criterion independently and returns a verdict with a short rationale
  • GEPA — prompt optimizer used to calibrate the judge against human labels
  • Conversation simulator — an LLM plays the shopper in scenarios with recorded tool fixtures
Continue reading — it's free

Google or a one-time email code. No password.