Selecting and monitoring physician annotators

Anton Krasnov4 Jun 2022

Problem

Celsus builds models on radiological studies, so doctors have to label the data. The labeling team has no radiology training and cannot judge a doctor's polygons itself. Three questions follow: where to find radiologists, how to check that they see pathology, and how to verify their work later. Staff doctors annotated slowly because labeling was not their main job and they had no reason to speed up.

Example from the article: doctors from Saint Petersburg almost never marked fibrocystic mastopathy on mammograms. Their local radiology school reads it differently, so the team formalized the labeling rules until everyone marked it the same way.

Solution in brief

Hire doctors on per-batch civil-law contracts, paid only after review. Admit them through a two-stage test on studies with known answers. Then control quality with automatic checks, two or more independent labels per study, and a visual review of about 20% of each dataset.

Questions this case answers
  • How to recruit and qualify expert annotators for a labeling project when the annotation team cannot judge the domain content itself
  • How to monitor the quality of expert labels at scale when experts disagree and manual review covers only a sample
  • How to find and screen doctors for annotating medical images
  • How to check that a candidate annotator actually sees pathology before paying for production work

Tools and methods

  • Two-stage entrance test — binary pathology call, then polygon annotation against reference
  • Cross-annotation — at least two doctors per study, conflicts analysed per doctor
  • Automatic checks — missing tags, self-intersecting polygons, size limits per class
  • Visual review — about 20% of each dataset
  • Notion — rules with videos and screenshots
Continue reading — it's free

Google or a one-time email code. No password.