Transformers for Personalization

Kirill Khrylchenko4 Jun 2022

Problem

Yandex search, search advertising and ad network rank documents or banners using each user's history of hundreds or thousands of queries and clicks. A cross-encoder that mixes the history with every candidate gives the best quality, but it recomputes all transformer layers for each candidate. Cost and latency grow with history length, so most of the history is dropped. Click labels are also distorted by position, popularity and trust.

Example from the article: in search advertising the system decides whether to show ads for a query, narrows a large ad set to a small one, and returns three final banners.

Solution in brief

Compress the history once per query into a few user vectors and score candidates with dot products against precomputed candidate vectors. Pretrain contrastively on billions of clicks, and move older history into an offline transformer whose output is stored per user. Train on a click probability split into position, relevance and bias parts.

Questions this case answers
  • How to personalize ranking with a long behavior history under a strict real-time latency and compute budget
  • How to learn relevance from implicit feedback that is distorted by position, popularity and trust biases
  • How to use a user history of thousands of events in a ranking model when latency and compute are tight?
  • How to turn a slow cross-encoder personalization model into a bi-encoder without losing most of the gain?

Tools and methods

  • Cross-encoder with early fusion — baseline: one transformer over everything.
  • Late fusion — history transformer once per query, small candidate transformer.
  • Bi-encoder (two-tower) — 6 user vectors against one candidate vector via dot product.
  • Contrastive pretraining with in-batch negatives — makes the bi-encoder beat the early-fusion model.
  • Offline plus real-time transformer — long history compressed offline, fresh events live.
  • Click decomposition — separate predictions for seeing, relevance, conformity, popularity.
Continue reading — it's free

Google or a one-time email code. No password.