Problem
Teams shipping LLM features cannot test them like ordinary software. Outputs are non-deterministic, "correct" is subjective, and one request can chain retrieval, reasoning, tool calls and generation, each failing on its own. Without a plan, three things happen: a generic "helpfulness" score looks good while the real failure goes unmeasured, a prompt change quietly degrades an unmeasured dimension, and effort goes into eval pipelines whose metrics do not track outcomes.
Example from the article: a travel-support assistant is prototyped, 100 outputs are read by hand, and the failures found (unsupported policy details, verbosity, over-refusal, broken JSON) become the first evals.
Solution in brief
Start by reading outputs and categorizing failures, then encode each failure type as a cheap programmatic check or a narrow LLM judge. Calibrate every judge against a human-labeled golden set that includes bad examples, and keep human review for edge cases. Run the same checks on sampled production traffic.
Questions this case answers
- How to measure the quality of a system whose outputs are non-deterministic and where correctness is subjective
- How to make an automated grader agree with human judgment well enough to replace most manual review
- How to evaluate a multi-step pipeline of retrieval, reasoning and tool calls component by component
- How to decide whether an LLM-powered product feature is good enough to ship?
Tools and methods
- LLM-as-a-judge ("virtual judge") — a stronger LLM scores outputs against a one-dimension rubric
- Golden set — 50-100 expert-labeled examples, bad ones included, used to calibrate judges
- Programmatic checks — schema and length checks that run before any judge or human
- Human evaluation — ground truth, high-stakes cases, resolving judge disagreements
- Trace-based agent evaluation — rebuild the trace tree to check subagents and tool calls
Google or a one-time email code. No password.