On the Deep Value of Useless Email Campaigns

Maria Anisimova24 May 2025

Problem

Mail's antispam ML team wanted personalization for the email service: personal spam filtering, interest profiles for neighboring services, and highlighting important messages. Receipts and orders describe users well but raise ethical and legal questions. Newsletters are a safer signal, yet users treat their folder as a dump, so important messages get lost. The first version of the highlighting feature flagged nearly everything for some users, even though offline metrics looked excellent.

Example from the article: a shared cosine threshold on the user and email embeddings gave 80% precision and 40% reads offline. Colleagues who tried the feature reported that emails were highlighted for everyone. For 1% of users practically the entire mailbox was highlighted, because they read every newsletter.

Solution in brief

Train a universal email text embedding, then a transformer that turns a user's newsletter history (reads, flags, categories) into a personal embedding. Score each incoming email with a separate model on top of that embedding plus extra features such as sender reputation, instead of a single cosine threshold. Validate in shadow mode before showing any icon.

Questions this case answers
  • How to build a personal user embedding from a weak, low-engagement signal and use it to highlight the items that matter for each user in a cluttered stream
  • How to roll out a per-user highlighting feature in shadow mode so that a single shared similarity threshold does not flag almost everything for heavy users
  • How to build a personal user embedding from a weak signal such as newsletter reads
  • How to highlight the emails that matter to a particular user in a low-engagement folder

Tools and methods

  • Universal text embedding — a BERT trained on email data with several tasks; distilled from 12 to 3 layers
  • User embedding — a transformer over email embeddings plus encoded user signals
  • Scoring model — trained on top of the embedding with additional features
  • Shadow mode — online scoring with nothing shown to users
Continue reading — it's free

Google or a one-time email code. No password.