Problem
Zepto, an Indian quick-commerce app with 8-12 minute delivery, saw support tickets grow faster than orders (which grew 100% year on year). It built an AI support agent stack: an intent router, specialized agents and image-based damage scoring, handling 80,000 tickets a day. Any prompt, model or orchestration change could silently break something. Quality was judged by anecdotes and social media, and human raters disagreed on subjective cases such as how rotten a product looks.
Example from the article: for the where-is-my-order agent, token count per ticket correlated negatively with customer satisfaction. Traces showed tickets looping on a stale delivery estimate while orders waited for a rider.
Solution in brief
Treat evaluation as infrastructure with two loops. In development, every agent version is scored on a golden dataset and must pass a stakeholder-defined quality gate. In production, sampled traces go to LLM judges and rule-based scorers, alerts fire on failures, and failures flow back into the golden dataset.
Questions this case answers
- How to decide automatically whether a new version of an LLM agent may go to production, using a golden set and a quality gate agreed in advance by several business functions
- How to calibrate an LLM judge against human raters when the labels are subjective and the raters disagree
- How to evaluate a production LLM support agent so quality is measured, not anecdotal
- How to decide automatically whether a new agent version (prompt, model or logic change) may go to production
Tools and methods
- MLflow 3.0 — tracing, scorers, evaluation pipeline, prompt optimizers
- DSPy — prompt optimization and few-shot selection against the team's metric
- LLM-as-a-judge and rule-based scorers — intent, groundedness, tone, schema, latency, safety
- Golden dataset — expectations from several stakeholders
- Cohen's kappa and MAE — agreement of humans and AI
- Stratified trace sampling, topic modeling — find weak segments and pain points
Google or a one-time email code. No password.
