Problem
Spotify serves recommendations from a large personalized language model instead of separate retrieval and ranking systems per surface. vLLM, the engine of the first launch, runs beam search outside the GPU core. This adds CPU bottlenecks and recomputes partly filled KV blocks, so the team fell back to cheaper accelerators. Ranking and personalization needed two more features that no single engine had.
Example from the article: the January 2026 podcast recommender decodes semantic IDs with beam search and maps the top sequences to catalog items.
Solution in brief
Spotify and NVIDIA closed the feature gaps in TensorRT LLM (TRT-LLM): beam search in the sampler, multi-item scoring and soft-token injection through a custom Qwen3 plugin. For personalization, a learned projection turns an existing user embedding into one soft token in the prompt. On the 1.7B Qwen3 model this cuts cost by up to 54% for beam search and up to 7% for scoring.
Questions this case answers
- How to serve a generative retrieval model that decodes item-code sequences with beam search under a strict latency budget and a limited GPU budget
- How to score hundreds of candidates with a language model in one request while keeping compute linear in the number of candidates
- How to condition a language model on a precomputed dense user vector without lengthening the prompt
- How to serve LLM-based recommendations (beam search over item tokens) at lower GPU cost?
Tools and methods
- Beam search over semantic IDs — candidate generation: the model decodes item-code sequences and the top beams are mapped to catalog items
- Multi-item scoring — rank many candidates by log probabilities in one request
- Soft token personalization (Embedding to Prefix) — project a user embedding into the LLM embedding space as one token
- TensorRT-LLM — the engine that unifies the three features
- vLLM, SGLang — baseline engines for beam search and scoring
- Qwen3 (1.7B, 4B) — benchmarked models
Google or a one-time email code. No password.