Problem
Computer vision teams overspend on image labeling and miss deadlines. Onboarding a new client often needs an extra dataset that matches the client's data. A power-line clearance monitor trained on Swiss data, for example, errs on Russian satellite images, at least because of different climate zones. Tech leads also named high onboarding cost and research into labeling approaches that does not always pay off.
Example from the article: a farmer-scoring segmentation task with expert labelers at $3 per unit. Active learning returns 2.5 rubles of saved labeling cost per ruble invested there, but does not pay off for mask-or-no-mask classification at 3 cents per unit.
Solution in brief
Train a model on a small labeled batch and pick the next images to label by an informativeness score: uncertainty, diversity, or both. A three-part check (labeling economics, model symptoms, task type) tells when it is worth the effort. Score recipes are given for classification, detection and segmentation.
Questions this case answers
- How to choose which unlabeled images to send to annotators so that a model reaches the target quality with the fewest labels
- How to decide whether active learning pays off given the per-item labeling cost and the type of vision task
- How to reduce image labeling cost in computer vision projects
- How to decide whether active learning pays off for a given labeling task
Tools and methods
- Active learning loop — label a random batch, train, score the unlabeled pool, label the top-scored batch, repeat.
- Uncertainty scores — least confidence, margin and entropy over class probabilities.
- Diversity scores — VAE reconstruction error; siamese embeddings with Gaussian mixture clustering.
- ROI formula — compare labeling cost saved with the cost of building the pipeline.
- Lightly, Encord Active, Baal, Modal — platforms and libraries that provide sampling methods.
Google or a one-time email code. No password.
