Problem
A recommendation request carries a long user history, around 16K tokens of past actions at Pinterest. That sequence is copied for every candidate item scored, hundreds to thousands of times per request, in storage, data loading and the forward pass. After a 100x growth in transformer dense parameters, storage, training and serving costs threatened to grow in proportion. Sorting data by request saves space but breaks the independent-and-identically-distributed (IID) assumption and lowers quality.
Example from the article: in the funnel, retrieval returns about 10K candidates, lightweight ranking keeps about 1K, full ranking keeps about 100, and the same user sequence is attached to each candidate at every stage.
Solution in brief
Store and process request-level data once per request, not once per item. Sort training rows by request so columnar compression removes the copies, then fix the two side effects: Batch Normalization statistics and false negatives in contrastive loss. Share the user computation across candidates: the user tower in retrieval, a cached key-value context in ranking.
Questions this case answers
- How to avoid repeating per-request computation and storage for every candidate when a large shared context is attached to many rows
- How to keep model quality when batches are no longer sampled independently and identically distributed
- How to cut training and serving cost of a large model by sharing the encoded context between its candidates
- How to stop storing and recomputing the same long user history once per candidate item in recommendation training and serving?
Tools and methods
- Request-sorted Apache Iceberg tables — compress the repeated user sequence
- SyncBatchNorm — normalization statistics across all devices
- User-level masking — drop in-batch negatives from the anchor's user
- Deduplicated two-tower forward pass — run the user tower once per unique request
- Deduplicated Cross-Attention Transformer (DCAT) — candidates cross-attend to a cached user context
Google or a one-time email code. No password.