Eval-driven development: Lessons from evaluating GenAI at scale

Rohit Girme, Dan Miller, Mia Zhao, Lifan Yang, Clint Kelly26 Aug 2026

Problem

Teams shipping LLM features cannot test them like ordinary software. Outputs are non-deterministic, "correct" is subjective, and one request can chain retrieval, reasoning, tool calls and generation, each failing on its own. Without a plan, three things happen: a generic "helpfulness" score looks good while the real failure goes unmeasured, a prompt change quietly degrades an unmeasured dimension, and effort goes into eval pipelines whose metrics do not track outcomes.

Example from the article: a travel-support assistant is prototyped, 100 outputs are read by hand, and the failures found (unsupported policy details, verbosity, over-refusal, broken JSON) become the first evals.

Solution in brief

Start by reading outputs and categorizing failures, then encode each failure type as a cheap programmatic check or a narrow LLM judge. Calibrate every judge against a human-labeled golden set that includes bad examples, and keep human review for edge cases. Run the same checks on sampled production traffic.

Questions this case answers
  • How to measure the quality of a system whose outputs are non-deterministic and where correctness is subjective
  • How to make an automated grader agree with human judgment well enough to replace most manual review
  • How to evaluate a multi-step pipeline of retrieval, reasoning and tool calls component by component
  • How to decide whether an LLM-powered product feature is good enough to ship?

Tools and methods

  • LLM-as-a-judge ("virtual judge") — a stronger LLM scores outputs against a one-dimension rubric
  • Golden set — 50-100 expert-labeled examples, bad ones included, used to calibrate judges
  • Programmatic checks — schema and length checks that run before any judge or human
  • Human evaluation — ground truth, high-stakes cases, resolving judge disagreements
  • Trace-based agent evaluation — rebuild the trace tree to check subagents and tool calls
Continue reading — it's free

Google or a one-time email code. No password.