Problem
An ML team ordered labeling in huge, rare batches from freelance experts through a separate data team. Nobody who understood the domain checked quality. Error analysis of the trained model then showed low labeling quality, and the data was cleaned by hand.
The speaker's team builds a chest X-ray and fluorography model that detects 14 radiological signs. Doctors disagreed on how to read the signs. They also made careless mistakes under volume pressure.
Example from the article: in 2022 the team spent 6 million on labeling and the metric rose from 0.838 to 0.874 (3.6 points).
Solution in brief
Treat labeling as a managed process, not a purchase. Doctors label small batches, a domain-expert supervisor reviews every batch and gives written feedback, and automated tests reject invalid labels the moment a doctor submits them. Work is requested monthly in small volumes, and the team keeps only the doctors who pass a quality check.
Questions this case answers
- How to get consistent labels from many remote domain experts working in small batches, when disagreements come from misunderstanding of features and from careless work
- How to block invalid labels automatically at submission time instead of discovering them later in error analysis after training
- How to get consistent, high-quality labels from freelance domain experts
- How to cut the cost of labeling while the model metric still improves
Tools and methods
- Supervisor review loop — each 100-study batch is reviewed by a senior radiologist who logs errors and writes feedback.
- Automated labeling tests — a bot checks submitted jobs and rejects invalid ones.
- Extended rulebook — features, ML-side labeling advice, typical errors, an atlas of drawn examples.
- Expert leaderboard — sample-based scoring of expertise and diligence per doctor.
- Supervisely, Telegram bot, Telegram channel, Notion — labeling and review tool, test runner, doctor chat, error log.
Google or a one-time email code. No password.
