Problem
Pinterest's ads team predicts click-through rate (CTR) with a deep neural network. A model without real-time behavior features cannot adapt to what the user is doing now. Real-time behavior features make the model larger and more expensive to serve. They also tie it to fresh signals, so a stale or missing feature can push Batch Norm into producing huge values.
Example from the article: a production DCNv2 ranking model was extended with user action sequences on HomeFeed, RelatedPins and Search.
Solution in brief
Build sequence features from the user's recent engaged pins, ads and searches, and encode them with a Transformer for long-term interest plus a small DIN-style attention over the latest 8 events for short-term interest. Serve the larger layers in bfloat16 to cut inference time by 30%. Replace Batch Norm with Resilient Batch Norm, which skips running-variance updates for degenerate columns.
Questions this case answers
- How to add long and short user behavior histories to a click-prediction ranking model so that it adapts to recent interests
- How to keep inference cost and training stability under control when a ranking model grows in size and depends on real-time features
- How to make ads click-through rate predictions use a user's recent behavior?
- How to model a long sequence of user events in a ranking model without losing early feature interactions?
Tools and methods
- User sequence features — engaged events with GraphSage item embeddings
- Transformer encoder — long-term interest from the full sequence
- Deep Interest Network (DIN)-style attention — short-term interest from the latest 8 events
- Period sum pooling — sum pooling per period of the sequence
- bfloat16 mixed-precision inference — faster serving on tensor-core GPUs
- Resilient Batch Norm — Batch Norm that ignores abnormal variance updates
- PyTorch — model and precision types
Google or a one-time email code. No password.