Problem
Teams that train and validate models on medical CT and MRI need datasets they can trust, because the models affect diagnosis and treatment choice. Manual annotation always lets noise in: labeling slips, processing errors, misread instructions and genuine disagreement between doctors on ambiguous images. Expert annotation is expensive, the datasets are large, and re-annotating everything is not an option. The team needs to find the wrong annotations cheaply and send each one to the cheapest person able to fix it.
Example from the article: one annotator marked about 300 individual calcifications on a single image, where the instructions asked for a polygon around the affected area.
Solution in brief
Control the process at three stages: write detailed instructions with examples after studying the domain, monitor simple object statistics while labeling runs, and clean the finished dataset. Cleaning is a three-step loop: flag outliers with handcrafted features, an intermediate model and model or annotator disagreement, sort the flagged cases by the expertise needed to fix them, then fix and version the result.
Questions this case answers
- How to find and fix label errors in a large expert-annotated dataset without re-annotating all of it
- How to monitor annotation quality during labeling when the ML team cannot judge the domain content itself
- How to find and fix label errors in a large manually annotated medical imaging dataset without re-annotating everything?
- How to write annotation instructions that doctors follow correctly?
Tools and methods
- Handcrafted object features — size, count, intensity, location and polygon points as outlier detectors.
- Intermediate error-finding model — ranks suspicious objects in a large dataset.
- Selective visual review — experts check only flagged cases.
- Cross-annotation comparison — disagreement between doctors marks errors.
- Annotation versioning — keeps revisions apart.
- 3D Slicer, ITK-SNAP — annotation software for 3D data.
Google or a one-time email code. No password.
