Problem
Yandex ranks banners in its ad network and in search ads, where little is known about each user. A transformer over the user's event history ranks well only if the history is long. The two-tower setup squeezes the history into one vector, which caps quality. Letting candidates attend to the history (early fusion) lifts quality, but a transformer over about 8000 events is too slow at runtime. Production could afford only about 200 events.
Example from the article: on 8192 events, early-fusion Target-Aware ARGUS gave +2.1% log-likelihood gain (llp) against +1.1% for the two-tower model, but could not be deployed.
Solution in brief
X-split ARGUS splits the history in two. An offline encoder reads the long history (8192 events) once a day and leaves 8 embeddings per user. A runtime encoder reads the latest 256 events. A small cross-attention layer lets about 300 candidates look at both. Distillation, smoothed negatives and chronological training bring the offline gain to +1.8% llp.
Questions this case answers
- How to rank candidate items for a user from a long interaction history when serving latency allows only a short window of the most recent events
- How to combine long-history user signals with candidate-aware ranking when the expensive history encoder cannot run once per candidate at request time
- How to rank ads with a sequential transformer model that uses long user history under strict runtime latency limits
- How to move from a two-tower (late fusion) model to candidate-aware (early fusion) ranking without recomputing a long-history transformer per candidate
Tools and methods
- ARGUS — autoregressive transformer over tokenized user events, with a context token per hit
- Cross-attention layer — early fusion of candidates with the compressed history
- Windowed attention — limits the runtime encoder's context during one-pass training
- Logit distillation (L1) — Target-Aware ARGUS as teacher, training-only
- Chronological training — time chunks with target decay
Google or a one-time email code. No password.
