Problem
A mailbox gets hundreds or thousands of new messages a week. Most are clutter, but a few matter, and the user has to dig them out and read each one. The Yandex 360 team wanted to mark the important emails and show a short summary of each. Two things stood in the way. There is no explicit "important" label, and the intuitive rule "fines are important, marketplace promos are not" fails: about a third of boarding-pass recipients and about half of Afisha-ticket recipients ignore those emails. Training an LLM on customer mail was also off the table.
Example from the article: some users read, click through and probably review every marketplace email, while others never open one.
Solution in brief
Label importance by what users do with emails. Train a CatBoost model on text embeddings (DSSM), rules, counters and per-user features, where the user features carry most of the signal. For summaries, tune a prompt first, then fine-tune YaGPT on employee mailboxes with assessor corrections. Distill the result into a smaller model using the best of five teacher summaries per email.
Questions this case answers
- How to label item importance from users' implicit actions and predict it per user, when no explicit importance label exists and importance differs between users
- How to distill a fine-tuned LLM summarizer into a smaller model when training data from users may not be used
- How to detect which emails or notifications are important to a user from implicit behavior signals
- How to build a personalized importance classifier that combines text embeddings, rules, counters and user features
Tools and methods
- DSSM with contrastive loss — turns email text parts into embeddings
- CatBoost — importance classifier over about 1000 selected features
- YaGPT fine-tuning and distillation — email summarizer, 5x faster inference after distillation
- ChatGPT-4 Turbo with a tuned prompt — quality baseline
- Assessors (Good / Norm / Crit) — rate and correct summaries
Google or a one-time email code. No password.
