Building Gavel: AI evaluation infrastructure across Eternal

Anshuman Mishra, Ritick Bajpai, Anjali Chennupati, Vatsal Agarwal, Soumik Rakshit, Ravikiran S, Yash Tripathi, Rishab Singla, Shashank Chaudhary5 Oct 2026

Problem

When generative AI produces ten or a hundred times more output, the bottleneck becomes knowing whether the output is any good. Humans cannot keep pace, and generic scores such as "helpfulness" say nothing about the application. Eternal (Zomato, Blinkit, District) hit this with about 35,000 generated images, then with chatbot transcripts and voicebot calls. Reviewing one voicebot call by hand cost roughly ₹450.

Example from the article: the delivery-partner voicebot failed in ways generic metrics never show: silent periods, repeated responses, a misunderstood partner concern, hallucinated instructions.

Solution in brief

Read real outputs, name the failure modes, and turn each into a rubric that asks one question and answers pass or fail with a written critique. Let an LLM apply the rubrics at scale, but treat it as a classifier: align it with a domain expert on human labels and report precision and recall. Track token spend per rubric and move easy ones to cheaper models.

Questions this case answers
  • How to measure the quality of a generative system at a volume humans cannot review, so that the measurement can be trusted
  • How to align an automated judge with a domain expert and report its agreement when failures are rare
  • How to keep the per-item cost of automated evaluation low when many checks run on every item
  • How to check thousands of voicebot calls or chatbot conversations a day when human review is too slow and costly?

Tools and methods

  • LLM-as-a-judge — applies pass/fail rubrics to every call.
  • Human labeling by a domain expert — error analysis and labels for aligning the judge.
  • Few-shot examples from critiques — written critiques of human verdicts feed the judge prompt.
  • Simulated customer agent — a persona-driven LLM talks to the bot under test.
  • Multi-model verification — several vision-language models read one image, a judge aggregates.
  • Knowledge-base grounding — checks answers against supplied documents.
Continue reading — it's free

Google or a one-time email code. No password.