Problem
Yandex Advertising ranks ads for search and for sites of the Advertising Network (RSYA). Longer user histories improve quality: earlier transformer models saw 256 events, roughly two weeks, while the speaker equates 8192 events to a year of history. But the system peaks at 300k+ requests per second, and the model must answer within tens of milliseconds. Self-attention costs in history length, so a vanilla transformer cannot run over thousands of events on every request.
Example from the article: the goal is to take up to 8192 history events into account at a peak of 300k+ requests per second with a model latency budget of tens of milliseconds.
Solution in brief
Split the transformer in two. An offline part reads the full 8192-event history once a day and stores the result per user. A runtime part reads only the latest 100 events together with the current request context and is combined with the stored offline output. This kept about two thirds of the offline gain of a full runtime transformer at a fraction of the compute.
Questions this case answers
- How to use a very long per-user event history in a transformer ranker that must answer within tens of milliseconds at hundreds of thousands of requests per second
- How to split a sequence model into an offline part computed once per user and a light runtime part that still sees the current request context
- How to use a year of user history (thousands of events) in a transformer-based ads ranker under a latency budget of tens of milliseconds
- How to serve a heavy sequential user model at hundreds of thousands of requests per second
Tools and methods
- Split-ContextAware transformer — offline NoAware part plus runtime ContextAware part
- Autoregressive training with a causal mask — one pass yields hidden states for every event of a user
- Next Item Prediction and Feedback Prediction — the two pretraining task families
- Sliding window attention — limits the runtime receptive field
- Position debiasing — removes ad position bias during training
Google or a one-time email code. No password.
