Problem
A music service with a large catalog and hundreds of billions of listens still builds recommendations from short histories and small models. Public transformer recommenders use the last 100 events, 2 to 4 layers and a few million parameters. Models that Yandex used earlier saw 1,000 to 1,500 events, and longer histories gave nothing. The team wanted to know whether scale works for recommendations as it does for language models.
Example from the article: Yandex Music, 300 billion listens for training, and the goal of using the last 8,000 listened tracks per user.
Solution in brief
Argus represents each interaction as a (context, item, feedback) triple and pretrains a causal transformer on two tasks: predict the next item and predict the feedback on it. The pretrained model is fine-tuned as a two-tower ranker whose score becomes a feature of the main ranking model. User embeddings are computed offline, so the service only takes a dot product.
Questions this case answers
- How to scale a sequential user-history transformer over very long histories of thousands of events and pretrain it on both positive and negative feedback
- How to deploy a heavy history model in a ranking service that cannot run it online by precomputing user embeddings offline
- How to scale a sequential recommender transformer over long user histories (thousands of events)
- How to pretrain a user-history transformer so that it also learns from negative feedback and organic listening
Tools and methods
- Argus — pretraining with next item prediction and feedback prediction over (context, item, feedback) triples
- Causal transformer — one pass over the whole history yields predictions for every moment
- Sampled softmax — loss over a catalog of billions of items
- Two-tower ranking — fine-tuned model whose score feeds the gradient-boosting ranker
- Unified Embeddings (Google) — compact embedding matrices for a very large track catalog
Google or a one-time email code. No password.
