Problem
The data center at MTS AI labels audio, text and intents for ASR, TTS and bots. Customers want results fast and cheap, but tasks are complex: multi-page instructions, 300+ intent classes, entities that need normalization. Scaling the team is not the only way to meet a tight deadline. The team looked for ways to speed up each task without lowering quality.
Example from the article: a two-channel audio task needed segmentation, transcription, tags, NER and normalization. The team cut it by 15–17% with pre-labeling. For a 300+ class intent task, clustering, an embedded reference table and dynamic overlap gave 20–25%.
Solution in brief
Optimize three things: the task (clarify the instruction, split into stages), the process (verify only disagreements and weak annotators, stop overlap once consensus is reached) and the tool (pre-labeling, an embedded reference, hotkeys). Each step gave a measured speedup of a few percent, which adds up.
Questions this case answers
- How to cut human labeling cost and time without lowering label quality by verifying only disagreements and starting from pre-labels
- How to speed up labeling into hundreds of classes using a reference table and adaptive overlap between annotators
- How to cut data-labeling cost and time without losing label quality
- How to reduce the share of labels sent to moderation without lowering quality
Tools and methods
- Pre-labeling — WhisperX segmentation, two ASR models for transcription, automatic NER and normalization.
- Dynamic overlap with consensus — stop collecting answers once enough annotators agree.
- Fine-tuned verification — check disagreements and selected annotators, not a random half.
- Clustering and an embedded intent reference — narrow 300 classes to about 50 and search them in the tool.
- LLM as an annotator — one of the overlap votes (being tested).
- Label Studio — labeling tool.
Google or a one-time email code. No password.