Problem
ML features are scattered across operational databases, warehouse tables, event streams and per-team pipelines. Each new model starts with finding the signals and writing one-off extraction code, and the next team repeats the work. Copying everything into one central store would only add another silo to keep in sync.
Example from the article: the feature deal_amount_changes_30d (how many times a deal's amount changed in 30 days) must be available for both training and real-time inference.
Solution in brief
A registry records where each feature comes from. Features are ingested into the store or read in place from the CRM, a team's service or a team's Iceberg table. Online values live in HBase, history lives in Iceberg as wide point-in-time rows, and a loader syncs the two.
Questions this case answers
- How to give many teams one retrieval interface for features that live in different systems, without duplicating data or keeping copies in sync
- How to store feature history so that a training row sees only values that existed at its timestamp, even when events arrive late
- How to let many teams discover and reuse the same ML features instead of rebuilding them for every model?
- How to serve features from operational systems without copying them into yet another store?
Tools and methods
- Feature registry (MySQL/Vitess) — catalog of definitions, online and offline source, description, tags and owners for each feature.
- Online store (Apache HBase) — latest value per feature for low-latency inference.
- Offline store (Apache Iceberg on Amazon S3) — full feature history for training and batch work.
- Apache Spark — streaming job that stages online updates; batch libraries for offline inference and backfills.
- OpenMetadata — data catalog where features appear with definitions, ownership and lineage.
- Point-in-time rows — one wide row per entity and moment, so training needs no joins across feature tables.
Google or a one-time email code. No password.