Problem
Allegro, a large marketplace, files each offer in one of over 23,000 categories. A classifier trained on offers from well-defined categories moves offers out of vague ones such as Other and Accessories. The training labels are partly wrong: merchants pick the wrong category, similar categories overlap, and the category tree changes. The model memorises these errors and repeats them on new offers. Multi-annotator re-labeling is rarely feasible at this volume.
Example from the article: on 1.3M offers, 20% synthetic label noise cut test accuracy from 90.26% to 85.31%.
Solution in brief
Replace plain cross-entropy with a noise-robust training method. Compare candidates on a clean test set after corrupting a known share of training labels. Of seven methods, clipped cross-entropy won back almost all lost accuracy and lost only 0.04 p.p. on clean data.
Questions this case answers
- How to train a classifier that stays accurate when labels are wrong in an unknown share of cases and re-labeling at scale is too slow
- How to compare noise-robust training methods when real label errors are not known in advance
- How to train a classifier that stays accurate when a share of the training labels is wrong?
- How to choose a noise-robust training method for a text classifier with tens of thousands of classes?
Tools and methods
- Clipped Cross-Entropy (CCE) — caps the per-sample loss.
- Early Learning Regularisation (ELR) — keeps what was learned early, before noise is memorised.
- Jensen-Shannon Divergence loss (JSD) — a loss between cross-entropy and mean absolute error.
- Co-teaching (CT) — two networks pass each other low-loss samples.
- Self-Paced Learning (SPL) and Provably Robust Learning (PRL) — drop the batch share with the highest loss or gradient norm.
- Mixup — mixes pairs of samples and labels.
- Synthetic noise generator — corrupts a known share of labels to measure robustness.
Google or a one-time email code. No password.