Problem
Yelp lets users and business owners report reviews that break its content policy, and a User Operations team removes the violating ones. Yelp also wanted an automated filter for egregious cases (hate speech, lewdness, threats, personal attacks). Without high precision such a tool delays the evaluation of reviews. The task is hard because of sarcasm and metaphor, and because violations are very rare in real traffic.
Example from the article: in 2022, 26,500+ reported reviews were removed for threats, lewdness and hate speech; this history became the training data for a real-time classifier.
Solution in brief
Fine-tune a pretrained language model as a binary appropriate/inappropriate classifier on a dataset curated with moderators: severity scores, embedding-similarity expansion, sub-category resampling. Pick the threshold on simulated traffic with realistic violation rates, not on the balanced test set. Flagged reviews go to human moderators.
Questions this case answers
- How to train a binary text classifier for policy violations when labels come from past removals and the violating class is rare in production
- How to set a decision threshold for a rare-positive classifier when the test set is class-balanced and live prevalence is unknown
- How to detect hate speech, lewdness and threats in user reviews in real time with a fine-tuned language model?
- How to build a clean training set for a moderation classifier from moderator decisions and past removals?
Tools and methods
- Supervised fine-tuning of a pretrained LLM — binary classifier of reviews
- Moderator severity scoring — labels only for the most egregious and clearly clean reviews
- Sentence-embedding similarity — finds more reviews like the labeled ones
- Zero-shot and few-shot LLM classification — assigns sub-categories to unlabeled violations for resampling
- Silhouette score and t-SNE — checks class separation before training
- Mock-traffic simulation — sets the threshold at realistic prevalence
- Hugging Face, MLflow, Amazon S3 — model hub, registry, storage
Google or a one-time email code. No password.