Using predictive technology to foster constructive conversations

Cathleen Li3 May 2022

Problem

A social network wants to calm a conversation before it turns abusive, not after someone reports a comment. Abuse feeds on itself: at Nextdoor, 90% of abusive comments appear in a thread that already has another one, and 50% appear in a thread with 13 or more others. Checking a comment after it is written is too late. Scoring a thread before the next comment exists is hard because that comment is unknown and under 1% of comments are reported.

Example from the article: the thread score lets the product mute notifications on the triggering comment, remind a neighbor who is about to reply, or prompt the post author to close the discussion.

Solution in brief

Label a thread positive if a reported comment appears in it later. Embed the post and its comments with a multilingual-aligned BERT model and train a small dense classifier on the concatenated embeddings. Oversample positives by sampling posts first, then comments within each post. The model trained on U.S. data reaches AUC above 0.83 and carries over to several European languages.

Questions this case answers
  • How to predict a rare negative event in a sequence of user-generated texts before the next item appears, when only the history of the sequence is available
  • How to transfer a text classifier trained in one language to other languages when labeled data in those markets is scarce
  • How to predict that a comment thread will turn abusive before the abusive comment is written?
  • How to build a training set when under 1% of comments are reported and reports cluster in a few big threads?

Tools and methods

  • Reports as labels — the next reported comment in a thread is the positive label.
  • Two-step sampling — posts first, then positive and negative comments inside each post.
  • Multilingual-aligned BERT embeddings (sBert API) — text features that work across languages.
  • Dense neural layers — the classifier on concatenated embeddings.
Continue reading — it's free

Google or a one-time email code. No password.