Problem
Expert labeling of medical images is expensive, and labeling data at random wastes most of the budget on examples the model already handles. A deployed model keeps repeating the same errors, and some classes have almost no labeled examples. The team at Celsus needed a way to decide which studies doctors should label next, so that each labeled study moves the model's metrics as much as possible.
Example from the article: false positives for the class "fracture" on X-rays grew more frequent. They came from bone degenerative changes near the clavicle and from front and back ribs overlapping into a fracture-like shadow.
Solution in brief
Run a loop: the model predicts on the unlabeled pool, a selection strategy (uncertainty, diversity, or random as a baseline) picks the most useful studies, doctors label them, and the model is retrained. The talk shows four ways to drive this loop: a cheap surrogate model to choose the strategy, targeted selection of hard cases, doctor disagreement reports, and an ensemble of open-source models for rare classes.
Questions this case answers
- How to select the most useful unlabeled examples for expensive expert labeling when the labeling budget is limited
- How to compare data-selection strategies cheaply when retraining the production model on every iteration is too slow
- How to find examples of a rare class in a large unlabeled pool when the current model has too few labeled examples to rely on
- How to choose which medical images to send to doctors for labeling when the labeling budget is small?
Tools and methods
- Uncertainty sampling — pick examples the model is least sure about, near the decision boundary.
- Diversity sampling — pick examples unlike the current training set: cluster centroids, outliers, low logit activations.
- Random sampling — a share of data always chosen at random as the baseline.
- LightGBM — surrogate model on detector features for comparing strategies quickly.
Google or a one-time email code. No password.
