Problem
A support chatbot talks to customers in multi-turn sessions, and nobody can read enough of them to know whether a prompt or model change helped. Instacart needed a measurement that covers whole conversations, runs on logs at scale and still matches what a human reviewer would say. Without it, regressions in context handling or wasted turns go unnoticed and iteration is slow.
Example from the article: a user asked for a representative and then said "Continue where we left off". The bot repeated the same "specialists are available" message twice, and the evaluator flagged a context failure and an efficiency failure.
Solution in brief
Instacart built LACE (LLM-Assisted Chatbot Evaluation). Each chat session is scored against binary criteria in five dimensions by an LLM judge, and the judge is tuned against human ratings until they agree. Judge design is chosen per criterion: debate-style agents for simple and context-dependent criteria, a loose directional check for subjective ones.
Questions this case answers
- How to score multi-turn conversations of a generative assistant automatically at scale when only a small set of human ratings is available
- How to choose an LLM-judge design (single prompt, self-reflection, multi-role debate) for criteria of different difficulty
- How to measure the quality of a customer support chatbot across full multi-turn conversations without reading chats by hand?
- How to make an LLM judge agree with human reviewers on chatbot conversations?
Tools and methods
- LLM-as-a-judge — scores each session True/False per criterion and returns a rationale.
- Agentic debate — Customer, Support and Judge agents, each an LLM role, evaluate one chat.
- Self-reflection — the judge reviews and revises its own first verdict.
- Human rating loop — humans rate selected chats on the same criteria to tune criteria and prompts.
- Stratified sampling and dashboards — sample chats by topic and track scores and failed chats.
Google or a one-time email code. No password.