The 52x Multiplier: How Zepto Mastered AI Agent Evaluation at Scale

Deepak Dhankani, Gireesh Sreedhar K P18 Jun 2026

Problem

Zepto, an Indian quick-commerce app with 8-12 minute delivery, saw support tickets grow faster than orders (which grew 100% year on year). It built an AI support agent stack: an intent router, specialized agents and image-based damage scoring, handling 80,000 tickets a day. Any prompt, model or orchestration change could silently break something. Quality was judged by anecdotes and social media, and human raters disagreed on subjective cases such as how rotten a product looks.

Example from the article: for the where-is-my-order agent, token count per ticket correlated negatively with customer satisfaction. Traces showed tickets looping on a stale delivery estimate while orders waited for a rider.

Solution in brief

Treat evaluation as infrastructure with two loops. In development, every agent version is scored on a golden dataset and must pass a stakeholder-defined quality gate. In production, sampled traces go to LLM judges and rule-based scorers, alerts fire on failures, and failures flow back into the golden dataset.

Questions this case answers
  • How to decide automatically whether a new version of an LLM agent may go to production, using a golden set and a quality gate agreed in advance by several business functions
  • How to calibrate an LLM judge against human raters when the labels are subjective and the raters disagree
  • How to evaluate a production LLM support agent so quality is measured, not anecdotal
  • How to decide automatically whether a new agent version (prompt, model or logic change) may go to production

Tools and methods

  • MLflow 3.0 — tracing, scorers, evaluation pipeline, prompt optimizers
  • DSPy — prompt optimization and few-shot selection against the team's metric
  • LLM-as-a-judge and rule-based scorers — intent, groundedness, tone, schema, latency, safety
  • Golden dataset — expectations from several stakeholders
  • Cohen's kappa and MAE — agreement of humans and AI
  • Stratified trace sampling, topic modeling — find weak segments and pain points
Continue reading — it's free

Google or a one-time email code. No password.