Active Learning

Maria Garets4 Jun 2022

Problem

Expert labeling of medical images is expensive, and labeling data at random wastes most of the budget on examples the model already handles. A deployed model keeps repeating the same errors, and some classes have almost no labeled examples. The team at Celsus needed a way to decide which studies doctors should label next, so that each labeled study moves the model's metrics as much as possible.

Example from the article: false positives for the class "fracture" on X-rays grew more frequent. They came from bone degenerative changes near the clavicle and from front and back ribs overlapping into a fracture-like shadow.

Solution in brief

Run a loop: the model predicts on the unlabeled pool, a selection strategy (uncertainty, diversity, or random as a baseline) picks the most useful studies, doctors label them, and the model is retrained. The talk shows four ways to drive this loop: a cheap surrogate model to choose the strategy, targeted selection of hard cases, doctor disagreement reports, and an ensemble of open-source models for rare classes.

Questions this case answers
  • How to select the most useful unlabeled examples for expensive expert labeling when the labeling budget is limited
  • How to compare data-selection strategies cheaply when retraining the production model on every iteration is too slow
  • How to find examples of a rare class in a large unlabeled pool when the current model has too few labeled examples to rely on
  • How to choose which medical images to send to doctors for labeling when the labeling budget is small?

Tools and methods

  • Uncertainty sampling — pick examples the model is least sure about, near the decision boundary.
  • Diversity sampling — pick examples unlike the current training set: cluster centroids, outliers, low logit activations.
  • Random sampling — a share of data always chosen at random as the baseline.
  • LightGBM — surrogate model on detector features for comparing strategies quickly.
Continue reading — it's free

Google or a one-time email code. No password.