Problem
Teams that run transformer recommenders (SASRec, BERT4Rec) retrain them as often as daily, on one or two GPUs, over catalogs of hundreds of thousands to millions of items. Full cross-entropy over the whole catalog gives the best quality, but it builds a huge score matrix of sequence positions times items. Training becomes slow and the model does not fit into memory, so teams fall back to cheaper losses with fewer negatives.
Example from the article: batch size 256, sequence length 100 and a catalog of 1 million items in float32 need bytes, which the slide gives as 95.4 GB for the score matrix alone.
Solution in brief
Cut Cross Entropy (CCE) computes the loss block by block without materializing the score matrix, and skips the backward pass for blocks where softmax is below a threshold, because about 90% of gradients are near zero. The authors also add negative sampling to CCE and call it CCE-. On six datasets the speaker reports memory down by almost 10 times and training up to more than 2 times faster, with NDCG@10 about the same on most of them.
Questions this case answers
- How to train a sequential next-item model with a full softmax over a catalog of millions of items on one GPU without materializing the full score matrix
- How to skip gradient computation for softmax entries that are near zero and combine this with negative sampling
- How to train a transformer-based sequential recommender faster and with less GPU memory
- How to use full cross-entropy loss on a catalog of a million items without running out of GPU memory
Tools and methods
- Cut Cross Entropy (CCE) — block-wise cross-entropy that drops near-zero softmax gradients
- CCE- — CCE combined with negative sampling (the authors' new loss)
- SASRec, BERT4Rec — the transformer recommenders trained with these losses
- Triton kernels, Replay library — implementation of CCE and CCE-
Google or a one-time email code. No password.
