Trust no one, not even your training data! Machine learning from noisy data

18 Apr 2023

Problem

Allegro, a large marketplace, files each offer in one of over 23,000 categories. A classifier trained on offers from well-defined categories moves offers out of vague ones such as Other and Accessories. The training labels are partly wrong: merchants pick the wrong category, similar categories overlap, and the category tree changes. The model memorises these errors and repeats them on new offers. Multi-annotator re-labeling is rarely feasible at this volume.

Example from the article: on 1.3M offers, 20% synthetic label noise cut test accuracy from 90.26% to 85.31%.

Solution in brief

Replace plain cross-entropy with a noise-robust training method. Compare candidates on a clean test set after corrupting a known share of training labels. Of seven methods, clipped cross-entropy won back almost all lost accuracy and lost only 0.04 p.p. on clean data.

Questions this case answers
  • How to train a classifier that stays accurate when labels are wrong in an unknown share of cases and re-labeling at scale is too slow
  • How to compare noise-robust training methods when real label errors are not known in advance
  • How to train a classifier that stays accurate when a share of the training labels is wrong?
  • How to choose a noise-robust training method for a text classifier with tens of thousands of classes?

Tools and methods

  • Clipped Cross-Entropy (CCE) — caps the per-sample loss.
  • Early Learning Regularisation (ELR) — keeps what was learned early, before noise is memorised.
  • Jensen-Shannon Divergence loss (JSD) — a loss between cross-entropy and mean absolute error.
  • Co-teaching (CT) — two networks pass each other low-loss samples.
  • Self-Paced Learning (SPL) and Provably Robust Learning (PRL) — drop the batch share with the highest loss or gradient norm.
  • Mixup — mixes pairs of samples and labels.
  • Synthetic noise generator — corrupts a known share of labels to measure robustness.
Continue reading — it's free

Google or a one-time email code. No password.