Problem
A team building a mammography detector gets labels from several radiologists, and the doctors often disagree on both where a finding is and what class it is. The causes are human error, different experience and schools, poor image quality and vague instructions. Clearer guidelines, quality tests and entrance exams shrink the gap but do not close it. Merging the labels naively hurts the model: a plain union of the doctors' boxes makes it produce many false positives.
Example from the article: one mammography image annotated by four doctors; a plain union of their four annotations contains many objects, and a network trained on it gives many false positives.
Solution in brief
Treat the annotator as an input. Add an embedding layer where each row encodes one annotator's style, feed the selected row into the detector during training, and choose the embedding to use at inference. The speaker calls it simple and elegant. It avoids the multi-head approach, which gets unwieldy at around 50 annotators.
Questions this case answers
- How to train an object detector on labels from several annotators who disagree on both location and class, without averaging away or discarding conflicting objects
- How to condition a detector on the annotator at inference time when no single annotation is the correct one
- How to train a detector on medical images when several doctors annotate the same image differently
- How to model each annotator's style instead of averaging or discarding conflicting labels
Tools and methods
- Cross-annotation — each image labeled by several doctors, each doctor labels only part of the dataset
- Annotator embedding — a learnable matrix with one row per annotator, concatenated to detector feature maps
- Multi-head network — one head per annotator, averaged for the final prediction
- ROC AUC — metric for comparing methods on mammography
Google or a one-time email code. No password.