Problem
Models trained on human, behavioral or model-made labels often hit a quality ceiling that more data or a better architecture does not lift. The speaker's examples come from Yandex: speech recognition (ASR), OCR and organization matching. Labels there come from crowd workers, user behavior and older model versions, and each source brings noise: cheating or tired workers, robots, misread clicks, logging bugs. The noise caps model quality.
Example from the article: for OCR labels collected through reCAPTCHA, majority vote with fixed overlap gave about 70% label quality, because many robots and auto-solvers were in the annotation stream.
Solution in brief
Simplify the annotation task, train and test the workers, then collect several answers per task and aggregate them. Pick the number of answers (static or dynamic overlap) and the merge rule for the task type. For behavioral and pseudo-labels, find the sessions where labels are worse and filter them out.
Questions this case answers
- How to get clean training labels from many noisy annotators and aggregate their answers when majority voting finds no majority
- How to estimate whether label noise, rather than the model, limits the achievable quality of a model
- How to get clean training labels from noisy crowdsourcing (Toloka) workers
- How to choose the number of annotators per task (static vs dynamic overlap) and aggregate their answers
Tools and methods
- Majority vote, Dawid-Skene, Whitehill (GLAD) — label aggregation; the last two reweight workers by agreement with the majority.
- Dynamic overlap — stop collecting answers once a rule or an ML model deems the label reliable.
- Exam, honeypot, golden set, post-acceptance — worker quality control.
- Phoneme-distance clustering — aggregation of free-text transcriptions.
- Pseudo-label filtering — drop model-made labels from sessions where behavior signals an error.
Google or a one-time email code. No password.
