Problem
Mail's antispam ML team wanted personalization for the email service: personal spam filtering, interest profiles for neighboring services, and highlighting important messages. Receipts and orders describe users well but raise ethical and legal questions. Newsletters are a safer signal, yet users treat their folder as a dump, so important messages get lost. The first version of the highlighting feature flagged nearly everything for some users, even though offline metrics looked excellent.
Example from the article: a shared cosine threshold on the user and email embeddings gave 80% precision and 40% reads offline. Colleagues who tried the feature reported that emails were highlighted for everyone. For 1% of users practically the entire mailbox was highlighted, because they read every newsletter.
Solution in brief
Train a universal email text embedding, then a transformer that turns a user's newsletter history (reads, flags, categories) into a personal embedding. Score each incoming email with a separate model on top of that embedding plus extra features such as sender reputation, instead of a single cosine threshold. Validate in shadow mode before showing any icon.
Questions this case answers
- How to build a personal user embedding from a weak, low-engagement signal and use it to highlight the items that matter for each user in a cluttered stream
- How to roll out a per-user highlighting feature in shadow mode so that a single shared similarity threshold does not flag almost everything for heavy users
- How to build a personal user embedding from a weak signal such as newsletter reads
- How to highlight the emails that matter to a particular user in a low-engagement folder
Tools and methods
- Universal text embedding — a BERT trained on email data with several tasks; distilled from 12 to 3 layers
- User embedding — a transformer over email embeddings plus encoded user signals
- Scoring model — trained on top of the embedding with additional features
- Shadow mode — online scoring with nothing shown to users
Google or a one-time email code. No password.