Shepherd: How Stripe adapted Chronon to scale ML feature development

Ben Mears15 Apr 2024

Problem

Stripe's fraud models, such as those behind Radar, run inside the payment path, so feature retrieval time adds directly to payment API latency. At the same time, fraud patterns change quickly, so feature values must reflect new events almost at once. For example, a spike in transactions at one business should update features that the model uses for other businesses. Engineers must also find features in hundreds of terabytes of raw data and ship them fast.

Example from the article: a feature for a fraud model can be the total number of charges a business processed over the last seven days.

Solution in brief

Stripe built Shepherd on Airbnb's open-source Chronon, which defines each feature once and computes it offline and online. To reach Stripe's scale, it added a Flink streaming job, pre-aggregated "tiles" in the key-value store, and two key-value stores split by write pattern.

Questions this case answers
  • How to serve model features in real time with low latency and freshness when each feature aggregates a large number of recent events
  • How to keep offline training features and online serving features consistent when they come from one definition but run on different engines
  • How to build a feature platform that serves fraud-model features with both low retrieval latency and fresh values?
  • How to define a feature once and compute it consistently for offline training and online serving?

Tools and methods

  • Chronon — feature definition in a Python and Spark SQL API, with offline, streaming and online components and consistency monitoring
  • Apache Flink — stateful streaming that updates feature values with low latency
  • Tiling — pre-aggregated feature values kept in Flink state and flushed to the store
  • Dual key-value store — a cheap write-once store plus a memcache-based store for frequent writes
  • Apache Airflow, Apache Iceberg, Apache Hive, Apache Spark — scheduling, warehouse writes, partitioned source tables and offline jobs
Continue reading — it's free

Google or a one-time email code. No password.