Problem
A ranking model at Pinterest (Homefeed) fine-tunes a large pretrained ID embedding table together with the rest of the network. Training has been limited to one epoch, because a second epoch hurts: the table memorizes repeated IDs instead of generalizing. Extending from one to two epochs cut the offline save metric by nearly 6%. The ceiling also blocks reuse of small fresh datasets and starves components that need longer training, such as vision-language models (VLMs). Longer runs leave GPUs idle on embedding communication.
Example from the article: Homefeed ranking fine-tunes on a 3-week window in a single epoch. The team wanted a shorter window with several epochs, to get a fresher pretrained backbone.
Solution in brief
Keep the embedding table frozen while the rest of the network trains, then let the table adapt only in the last epoch. In production this was combined with pipelined embedding lookups and less frequent optimizer updates, and gave +1.09% online saves.
Questions this case answers
- How to train a model with a huge sparse embedding table for several passes over the data when the table memorizes repeated examples and quality collapses
- How to hide cross-device embedding lookup and optimizer synchronization latency in sharded sparse training without hurting model quality
- How to fine-tune large ID embedding tables in a ranking model for more than one epoch without overfitting?
- How to reuse a short window of fresh training data several times instead of one pass over a long window?
Tools and methods
- Frozen embedding table — removes the overfitting; best simple baseline
- Delayed embedding update — table frozen for N−1 epochs, co-trained in the last one
- Gradient isolation — blocks selected loss paths from reaching the table
- Semi-synchronous pipeline — prefetches the next batch's embeddings during the current step
- Gradient accumulation — fewer optimizer broadcasts, with a scaled embedding learning rate
- PyTorch, CUDA — streams and events for overlap
Google or a one-time email code. No password.