Sequence learning: A paradigm shift for personalized ads recommendations

Sri Reddy, Habiya Beg, Arnold Overwijk, Sean O'Byrne19 Nov 2024

Problem

Meta's ads ranking relied on deep learning recommendation models (DLRMs) fed with thousands of hand-built sparse features. Each feature aggregates a user's actions over a time window. Aggregation throws away the order of events and which attributes occurred together in one event. It also depends on human intuition about useful patterns, and it spawns overlapping variants that raise compute and storage costs.

Example from the article: a feature such as "Facebook pages visited in the past M days with a visit count" becomes a list like [(Page-id1, 45), (Page-id2, 30), ...], so the order of visits is lost.

Solution in brief

Replace aggregated features with event-based features (EBFs): one object per event with its attributes and timestamp. An event model turns each event into an embedding, and a sequence model with attention pooling summarizes the sequence for the ad being ranked. Custom jagged-tensor kernels keep ranking within the latency budget.

Questions this case answers
  • How to replace manually aggregated user features with a model that learns directly from ordered event histories while keeping ranking latency within a production budget
  • How to scale a ranking model to longer, variable-length event sequences without a proportional growth in compute and memory
  • How to replace hand-engineered aggregated user features with models that learn from raw event sequences?
  • How to keep ranking latency acceptable when user histories have variable length and the model attends over them?

Tools and methods

  • Event-based features (EBFs) — a unified input format: event stream, sequence length, event information
  • Event model — builds one embedding per event from attributes plus a timestamp encoding
  • Event sequence model — attention pooling over event embeddings, keyed by the candidate ad
  • Jagged tensors and Jagged Flash Attention — handle variable-length histories efficiently on GPU
  • PyTorch — native jagged tensor support
  • Quantization and vector quantization — allow longer sequences and richer event embeddings
Continue reading — it's free

Google or a one-time email code. No password.