Complex and Urgent: Cutting Labeling Costs Without Losing Quality

Anna Druzhkina24 May 2025

Problem

The data center at MTS AI labels audio, text and intents for ASR, TTS and bots. Customers want results fast and cheap, but tasks are complex: multi-page instructions, 300+ intent classes, entities that need normalization. Scaling the team is not the only way to meet a tight deadline. The team looked for ways to speed up each task without lowering quality.

Example from the article: a two-channel audio task needed segmentation, transcription, tags, NER and normalization. The team cut it by 15–17% with pre-labeling. For a 300+ class intent task, clustering, an embedded reference table and dynamic overlap gave 20–25%.

Solution in brief

Optimize three things: the task (clarify the instruction, split into stages), the process (verify only disagreements and weak annotators, stop overlap once consensus is reached) and the tool (pre-labeling, an embedded reference, hotkeys). Each step gave a measured speedup of a few percent, which adds up.

Questions this case answers
  • How to cut human labeling cost and time without lowering label quality by verifying only disagreements and starting from pre-labels
  • How to speed up labeling into hundreds of classes using a reference table and adaptive overlap between annotators
  • How to cut data-labeling cost and time without losing label quality
  • How to reduce the share of labels sent to moderation without lowering quality

Tools and methods

  • Pre-labeling — WhisperX segmentation, two ASR models for transcription, automatic NER and normalization.
  • Dynamic overlap with consensus — stop collecting answers once enough annotators agree.
  • Fine-tuned verification — check disagreements and selected annotators, not a random half.
  • Clustering and an embedded intent reference — narrow 300 classes to about 50 and search them in the tool.
  • LLM as an annotator — one of the overlap votes (being tested).
  • Label Studio — labeling tool.
Continue reading — it's free

Google or a one-time email code. No password.