How Pinterest Built a Real‑Time Radar for Violative Content using AI

Faisal Farooq, Aravindh Manickavasagam, Attila Dobi8 Dec 2025

Problem

Trust and safety teams usually watch user reports confirmed by human reviewers. Reports miss a lot. Self-harm is under-reported because of stigma, people who look for harmful content do not report it, and rare categories produce too few reports to show a trend. Platform-wide human studies ran about every six months and needed at least two reviewers per item plus adjudication, so before-and-after checks of an intervention were slow.

Example from the article: Pinterest breaks prevalence down by policy area (Adult Content, with nudity separate from explicit sexual content) and by surface (Homefeed, Search, Related Pins).

Solution in brief

Measure prevalence: the share of all views on a day that went to policy-violating content. Sample impressions with probabilities that favor high-exposure, high-risk-score items, label the sample with a multimodal LLM, and re-weight by inverse inclusion probability so the estimate stays unbiased. Check the LLM against human labels, and publish a daily estimate with a 95% confidence interval.

Questions this case answers
  • How to estimate the exposure-weighted rate of a rare harmful event type every day when human labeling is too slow and expensive to run at scale
  • How to use a risk model to focus a labeling budget on likely positives while keeping the population-level estimate unbiased
  • How to measure the share of daily views that went to policy-violating content without waiting for user reports?
  • How to sample impressions so that rare violations are found without biasing the prevalence estimate?

Tools and methods

  • Weighted reservoir sampling — samples by impressions and risk score
  • Hansen–Hurwitz and Horvitz–Thompson ratio estimators — inverse-probability re-weighting for with-replacement and without-replacement designs
  • Multimodal LLM labeler — labels sampled content against policy prompts written with subject matter experts
  • Gold set and human validation queue — checks LLM decision quality before launch and over time
Continue reading — it's free

Google or a one-time email code. No password.