How our content abuse defense systems work to keep members safe

Sanket Modi31 Jan 2022

Problem

A platform that hosts user content must keep policy-violating posts, messages and images away from members. The defense has several layers (automatic removal, human review, member reports), and each layer makes errors. Counting removals alone does not show how much abuse gets through, whether the auto-filter deletes good content, or whether reviewers label consistently. Undetected violations cannot be tracked, so they have to be estimated.

Example from the article: LinkedIn's Trust & Safety team filters content within 300 milliseconds of creation. It reports that in the first half of 2021 about 66.3 million violating pieces were removed, 99.6% of them by automated defenses.

Solution in brief

Treat all violating content as a funnel with three parts: prevented at creation, detected by humans, and undetected. Measure each part with its own quality metric (precision for automation, audit-based quality score for reviewers). Estimate the undetected part by sending a sample of all content to human review, drawing the sample with stratification by classifier score so it needs far fewer labels than a random one.

Questions this case answers
  • How to estimate the prevalence of rare violations that no detector has flagged when exhaustive labeling is too expensive
  • How to measure the accuracy of a layered defense where automation and human reviewers both make errors
  • How to measure what share of policy-violating content is blocked, caught by reviewers, or never found?
  • How to estimate the amount of violating content nobody has detected without labeling everything?

Tools and methods

  • Violation funnel — splits violating content into prevented, detected and undetected.
  • Stratified sampling by classifier score — cheaper estimate of undetected content.
  • Human review of samples and audits — gives precision and reviewer quality score.
Continue reading — it's free

Google or a one-time email code. No password.