Problem
A team trains an image model on labeled data and gets mediocre quality. The first reaction is usually to change the architecture or the loss. Often the cause is in the data: wrong labels, duplicates between train and test, groups of cases the model has never seen, a gap between training data and production. Each of these has its own detection method, and a team that knows only a few keeps tuning the model.
Example from the article: a model that finds lung pathologies on the first try flags regions outside the lungs and confuses classes. The speaker's team fixes the data and keeps the model.
Solution in brief
The talk sorts data-centric methods into three groups: training data, data after training (applicability boundaries, shifts, outliers) and data management. For each problem it names a method or library, for example cleanlab for label errors and Domino for weak slices.
Questions this case answers
- How to improve a trained image model by finding wrong labels, duplicates across merged datasets and groups where the model fails, instead of changing the architecture
- How to estimate how much labeled training data a task needs
- How to improve model quality by fixing the training data instead of changing the model
- How to find wrong labels and garbage images in a labeled dataset
Tools and methods
- Confident learning (cleanlab) — finds likely wrong labels and garbage images from model probabilities
- Perceptual hashes, ORB/SIFT descriptors, metric learning — find duplicates when merging datasets
- Core set selection with a proxy network — picks a training subset using a small model
- Programmatic labeling (Snorkel) — expert-written heuristics replace manual labeling
- Domino — finds slices where the model works badly
- Isolation forest, autoencoders — outliers and anomalies
- DICOM tags, S3, PostgreSQL — leak control and data storage in the medical setting
Google or a one-time email code. No password.
