Speed Up Feature Engineering for Recommendation Systems

29 Sep 2022

Problem

Snap's ML teams spent much of their time on features for recommendation models. A new feature had to be implemented and logged in critical online services, then teams waited weeks for logs before training. ML engineers feared touching serving code, and delegating to infrastructure engineers added coordination cost. Teams built overlapping infrastructure and could hardly share features.

Example from the article: total views of a Discover snap by hour of day over six hours and 30 days, or the last 100 unique Discover snaps a user viewed on Sunday with view time above 1 second.

Solution in brief

Robusta, Snap's feature platform, limits features to associative and commutative aggregations and stores pre-aggregated blocks for a hierarchy of time ranges. Any sliding window is assembled from these blocks. A feature is declared in one YAML spec, and training features are rebuilt offline from the same blocks, capped by a logged "feature version pointer".

Questions this case answers
  • How to generate training features offline that match online values when windowed aggregates change every minute and storing every value per key is infeasible
  • How to define and serve thousands of time-window aggregates from one declarative spec without per-feature service deployments
  • How to test a new recommendation feature without waiting weeks for feature logs from production?
  • How to reproduce offline the exact sliding-window feature values the model saw at online scoring time?

Tools and methods

  • Associative and commutative aggregations — count, sum, approximate quantiles: mergeable in any order
  • Pre-aggregated time blocks — 5-minute, 1-hour, 12-hour blocks combined into any sliding window
  • Lambda architecture on Spark — streaming job for freshness, batch job for completeness
  • Apache Iceberg — stores blocks for offline generation and backfills
  • Aerospike — key-value store for online block lookup
  • Feature version pointer — logged at online fetch, used to rebuild the same values offline
Continue reading — it's free

Google or a one-time email code. No password.