SpectroBrain: Detecting Phone Spam with Semi-Supervised Learning

Winston Quock23 May 2023

Problem

Spammers call Zillow Rentals users to pitch tax debt relief or cryptocurrency investments, and the team must block them. Phone users cannot report spam, so labels are scarce and spam is a small share of answered calls. Transcribing voice messages and running the existing email spam classifier on the text gave about 50% false negatives. Up to 80% of calls start with a preamble (a voicemail greeting or an automated menu) that is unrelated to the message and confuses a text classifier.

Example from the article: from 530 stratified, manually labeled phone messages only 48 turned out to be spam.

Solution in brief

Use Whisper and the existing RoBERTa spam model as they are. Train only a model that finds where the preamble ends and cut the audio there before transcription. It combines a self-supervised audio encoder trained on unlabeled calls with three weak models (frame classifier, change-point proposer, segment classifier).

Questions this case answers
  • How to classify a rare harmful class in audio-derived text when only a few hundred labeled examples exist and users cannot report the harm
  • How to locate a single change point in a long sequence from a handful of labeled boundaries by combining a representation learned on unlabeled data with simple supervised models
  • How to detect spam in phone voice messages when almost no labeled spam calls exist?
  • How to stop voicemail greetings and phone menus from confusing a text spam classifier?

Tools and methods

  • Whisper — speech-to-text for the trimmed audio, used as is.
  • RoBERTa — existing text spam classifier (further pre-trained on internal email and SMS, fine-tuned on user-reported spam), used as is.
  • MFCC (via librosa) — audio features for the encoder.
  • CNN-Transformer encoder — self-supervised masked-prediction encoder trained on about 50K unlabeled recordings.
  • ruptures — dynamic-programming change-point search that proposes candidate switch points.
  • CatBoost — classifier that judges whether a whole score segment is voice.
Continue reading — it's free

Google or a one-time email code. No password.