Yelp’s AI pipeline for inappropriate language detection in reviews

Gouthami Senthamaraikkannan12 Mar 2024

Problem

Yelp lets users and business owners report reviews that break its content policy, and a User Operations team removes the violating ones. Yelp also wanted an automated filter for egregious cases (hate speech, lewdness, threats, personal attacks). Without high precision such a tool delays the evaluation of reviews. The task is hard because of sarcasm and metaphor, and because violations are very rare in real traffic.

Example from the article: in 2022, 26,500+ reported reviews were removed for threats, lewdness and hate speech; this history became the training data for a real-time classifier.

Solution in brief

Fine-tune a pretrained language model as a binary appropriate/inappropriate classifier on a dataset curated with moderators: severity scores, embedding-similarity expansion, sub-category resampling. Pick the threshold on simulated traffic with realistic violation rates, not on the balanced test set. Flagged reviews go to human moderators.

Questions this case answers
  • How to train a binary text classifier for policy violations when labels come from past removals and the violating class is rare in production
  • How to set a decision threshold for a rare-positive classifier when the test set is class-balanced and live prevalence is unknown
  • How to detect hate speech, lewdness and threats in user reviews in real time with a fine-tuned language model?
  • How to build a clean training set for a moderation classifier from moderator decisions and past removals?

Tools and methods

  • Supervised fine-tuning of a pretrained LLM — binary classifier of reviews
  • Moderator severity scoring — labels only for the most egregious and clearly clean reviews
  • Sentence-embedding similarity — finds more reviews like the labeled ones
  • Zero-shot and few-shot LLM classification — assigns sub-categories to unlabeled violations for resampling
  • Silhouette score and t-SNE — checks class separation before training
  • Mock-traffic simulation — sets the threshold at realistic prevalence
  • Hugging Face, MLflow, Amazon S3 — model hub, registry, storage
Continue reading — it's free

Google or a one-time email code. No password.