Catch Me If You Can: LLM Quality Control for Customer Service

Anton Udalov24 May 2025

Problem

Yandex Crowd runs support for 67 Yandex services: 4.5 million contacts per month, 4.5 thousand operators, three channels (chat, email, audio). Twenty reviewers check closed dialogues by hand and cover only 3%. The checklist has 12 categories and more than 80 error classes, so some important or critical classes show up only once or twice in several months. Each error needs a comment for the operator with a knowledge-base link.

Example from the article: the class "did not mention the user's situation" in the response-adaptation category. The operator pasted a template instead of adapting it.

Solution in brief

Replace the sample with a daily LLM pipeline built around about 50 prompts, with regexes and simple string matching wherever they are enough. The pipeline is validated on a golden set of 300-350 re-labeled dialogues per error class, scored on a 0/1/2 scale. Prompts are versioned like code, and a second LLM judges how close generated comments are to historical ones.

Questions this case answers
  • How to check every dialogue against a long checklist of error classes with an LLM pipeline instead of a small manual sample, when some classes occur only a few times a year
  • How to validate an LLM-based checker on a golden set and version its prompts so that regressions are caught before release
  • How to check 100% of customer support dialogues for errors instead of a 3% manual sample
  • How to build a golden set and a scoring scale for judging an LLM pipeline against human reviewers

Tools and methods

  • LLM pipeline (YandexGPT) — finds error classes and writes the operator comment
  • Golden set with a 0/1/2 scale — reviewers rate each model answer
  • Prompt version table — quality per version, mapping to error class, author comment
  • LLM as a judge (self-hosted DeepSeek) — similarity of two comments, 0 to 1
  • Regex, SequenceMatcher, Yandex Speller — cheap checks
Continue reading — it's free

Google or a one-time email code. No password.