Scaling Loss Functions for Sequential Recommendations with Large Item Catalogs

Danil Gusak24 May 2025

Problem

Teams that train transformer recommenders (SASRec and similar) with full cross-entropy over the item catalog hit a memory wall as the catalog grows. The loss needs a logit matrix of batch states times catalog items. Unlike in NLP, the model itself is small, so with a 1-million-item catalog the loss takes about 99.9% of GPU memory during training. The usual escape is negative sampling, which works poorly with few negatives and ignores hard negatives.

Example from the article: a batch of 64 sequences of 200 items over a 1-million-item catalog needs more than 100 GB of GPU memory, so it does not fit on one H100.

Solution in brief

Scalable Cross-Entropy (SCE) computes only the logits that matter for the gradient. Batch states and catalog embeddings are projected onto a small random matrix and sorted into buckets. Logits are computed only inside each bucket, so memory no longer depends on the catalog size.

Questions this case answers
  • How to train a sequential recommender with full softmax loss over a catalog of a million items when the loss matrix does not fit into GPU memory
  • How to cut peak memory and training time of a sequential recommender without losing ranking quality by computing only the logits that matter for the gradient
  • How to train a sequential recommender with full softmax loss on a catalog of a million items without running out of GPU memory
  • How to cut peak GPU memory and training time of a SASRec-style model without losing ranking quality

Tools and methods

  • Scalable Cross-Entropy (SCE) — approximate softmax loss from bucketed, truncated logits
  • SASRec — transformer decoder backbone for next item prediction
  • Negative sampling baselines (BCE with several negatives, gBCE, sampled cross-entropy) — comparison points
  • RePlay — Sber's recommender library; SCE was added to it shortly before the talk
Continue reading — it's free

Google or a one-time email code. No password.