Problem
Yandex Crowd is the internal labeling service of Yandex: more than 10 million tasks per month across hundreds of projects. It moves projects from human annotators to LLMs and multimodal VLMs (16% of actively labeled projects at the time of the talk). Fine-tuning a model per project is slow and expensive, and each project has little training data. Yet customers expect quality to hold for the whole life of the project, so the team needs a per-answer trust signal and a threshold that can be set on each new project without a DS specialist.
Example from the article: labeling of queries to the voice assistant Alice. Accuracy grew from 50% to 70%, and 21% of answers were confident enough to accept.
Solution in brief
Keep the model's answers unchanged and score only its confidence. Build it from several few-shot prompts that imitate different annotators: take the majority answer and its average token probability. Choose the threshold on 200–500 human verdicts from the same project, so the error on accepted answers stays below a target with a stated probability. The rest go to people.
Questions this case answers
- How to calibrate the confidence of a generative model's answers so that automatic labels meet a target error rate and the remaining items go to human annotators
- How to choose a confidence threshold from a few hundred human verdicts for a new labeling task without a data scientist
- How to calibrate LLM confidence so that automatic labels meet a target error rate
- How to decide which LLM labels can skip human annotators
Tools and methods
- Simulated annotators — N prompts with different few-shot examples
- Token-probability confidence — no extra model calls
- Threshold search on a calibration set — empirical risk plus a confidence interval
- LLM / VLM inference — YaGPT and YaVLM in production
Google or a one-time email code. No password.
