Problem
Avito checks more than 20 million listings a day with more than 2,000 rules, and over 95% of content is processed automatically. Each rule used to get its own model, built from scratch in a notebook, which took a week or more. Labels are noisy, with moderator agreement sometimes at 50/50. Violations are rare, rules change and users try to bypass them, so models degrade and the flow to manual review swings.
Example from the article: a model trained on a few dozen hand-collected positives produced a huge stream of false positives. After several labeling iterations, its precision grew from 5% to 46%.
Solution in brief
Relabel only the objects where out-of-fold predictions disagree most with the labels. Grow rare classes by labeling the top-scored triggers of a noisy model. Move thresholds automatically to hold precision or flow, and watch each model's share in the flow with a likelihood-based change-point test.
Questions this case answers
- How to train a classifier for a rare violation from a few dozen positives and noisy labels by relabeling the examples where out-of-fold predictions disagree most with the labels
- How to detect that a deployed classifier has started misbehaving by monitoring its share of the flow with a likelihood-based change-point test instead of a learned model
- How to keep content moderation models accurate when labels are noisy and rules change constantly
- How to train a classifier for a rare violation with only a few dozen positive examples
Tools and methods
- K-fold self-fix — relabel the largest label-vs-prediction disagreements
- Iterative training — label top N by proba, add to the dataset, retrain
- CLIP embeddings, BERT with LoRA adapters — shared features, small classifiers on top
- OpenCV, Aqueduct, TensorRT, NVIDIA MIG, LoRAX, FlashAttention — serving speed-ups
- Automatic thresholds — hold a precision or flow target in real time
- Change-point criterion on the model's share in the flow — alert within minutes
Google or a one-time email code. No password.