Problem
A marketplace with more than 100 million items, over 90 million active buyers and 7 million active sellers has to remove listings that break its policies. Etsy combines community reports, automated controls and machine learning. Millions of new listings arrive per day, so manual review cannot cover them. Almost every listing is clean, so a detector must catch rare violations without hurting well-intentioned sellers.
Example from the article: Etsy's Trust & Safety team trained one supervised classifier on listing text and images. It helped identify and remove more than 100,000 violations over the past year.
Solution in brief
Train one multimodal classifier with a "no violation" class next to the violation classes. Build the dataset from annotated positives, hard negatives and random easy negatives, and split it by time. Fight imbalance with focal loss. Judge the model offline with precision and a recall proxy, then canary it, A/B test it against the current champion and retrain when quality drops.
Questions this case answers
- How to detect rare policy-violating objects with text and image content when only a small, human-annotated sample of the stream can be labeled
- How to evaluate and promote a classifier when true recall cannot be measured on the full stream
- How to detect policy-violating listings automatically when millions of new listings arrive every day?
- How to build training data for a violation classifier when almost all listings are clean?
Tools and methods
- BERT, ALBERT — text encoder; ALBERT shares transformer parameters to cut compute.
- EfficientNet — image encoder.
- EmbraceNet-style late fusion — per-modality projection, concatenation, softmax.
- Focal loss — handles class imbalance.
- Hard and easy negatives — annotated non-violations plus random listings.
- Time-based split — train, validation and test ordered in time.
- Canary release, champion/challenger A/B test — rollout and promotion.
Google or a one-time email code. No password.