Problem
A team building ML for pathology needs labels that only physicians can produce. One expert hour costs about one data-scientist day, and the experts disagree with each other and with themselves. The speaker once counted annotation as 5% of the work and now puts it at 30%. Crowdsourcing does not work here: wrong labels cost far more than slow ones, and a fixed test set stops matching production.
Example from the article: five trained pathologists labeled the "tumor" class on the same slides, and every pair scored an IoU between 0.35 and 0.46.
Solution in brief
Treat the labeled data and the instruction as the product. Calibrate a few annotators on shared items before bulk labeling, and measure agreement between people and within one person. Version the instruction like code and store its version beside each label. Keep the golden set dynamic: model errors go to expert consensus, and agreed cases join the set.
Questions this case answers
- How to build a labeling and validation pipeline when only domain experts can label and their agreement with each other is low
- How to keep an evaluation set current when production data drifts and expert labels are costly
- How to build a labeling and validation pipeline when only domain experts can label and they disagree
- How to measure agreement between expert annotators and decide whether the instruction or the people are the problem
Tools and methods
- Annotator calibration — the same few items go to everyone; the instruction is revised until all read it alike.
- Inter-annotator agreement (IoU matrix, Cohen's kappa) — shows whether the instruction is ambiguous.
- Intra-annotator agreement — the same item shown again after augmentation.
- ROI and mosaic annotation — labeling gigapixel slides without pixel-level work on the whole image.
- Instruction versioning (SemVer) — changelog, edge-case tests and review.
- Dynamic golden set — grows through consensus on model errors.
Google or a one-time email code. No password.
