Phishing detection + domain experts

Igor Kabanov, Alsu Nurutdinova29 Oct 2022

Problem

A security-product ML team had to flag phishing emails in the full mail stream. Phishing is rare, looks almost identical to normal mail, and has to be caught in real time with at most 10 errors per day (false positives plus false negatives) on about 10 thousand emails. The model must run without GPUs and answer within 10 seconds. There were no per-user statistics and no domain lists. The labels and feature ideas came from experts, who could not always say why an email was phishing.

Example from the article: experts judged emails by attachment names and "just saw" phishing, but could not give a list, patterns or word combinations to build a feature from.

Solution in brief

Split the work into two branches, email structure (headers, metadata) and text, and run a weekly loop: experts propose hypotheses, the team turns them into features and checks the metrics. Every expert feature is tested on the team's own sample. Experts help most with clustered model errors, but their labels need checking too.

Questions this case answers
  • How to build a lightweight real-time classifier for a very rare, adversarial class when the only labels and feature ideas come from domain experts
  • How to turn expert intuition into measurable features and verify them against a labeled sample
  • How to detect phishing emails in a real-time mail flow when phishing is very rare?
  • How to turn domain experts' intuition into features a model can use?

Tools and methods

  • Header and metadata features — From/Return-Path mismatch, Received chain, DMARC result, attachment names.
  • Text features — email length, phishing domains, tone and call to action via zero-shot models.
  • Imbalanced learning — upsampling and downsampling for the rare phishing class.
  • Expert error review — clustered model errors sent to experts for labels and new hypotheses.
Continue reading — it's free

Google or a one-time email code. No password.