Problem
Pinterest trains recommendation and ads models on tens of petabytes of engagement logs spanning many months. To test a new feature, the default route is forward logging: add the feature to serving logs and wait for data. Each iteration then takes 3 to 6 months, touches training and serving code, and shares a logging pipeline with production, which can cause data loss incidents. Backfilling historical feature values avoids the wait. The first Spark version, though, was slow, expensive and could not run in parallel.
Example from the article: four engineers each backfilling one feature group into a 120-day training range needed a cumulative 140 days with the original backfiller.
Solution in brief
Split the backfill in two. Stage 1 joins each new feature group only with the ID columns of the training table and writes a small staging table. Stage 2 promotes many verified staging tables into the production table in one batch. Iceberg bucketing makes the joins cheap, and a Ray data loader can join staging tables at training time, so Stage 2 can be skipped for offline experiments.
Questions this case answers
- How to add a new input column to a very large historical training table quickly, cheaply and without touching the production copy
- How to let many people attach independent columns to one shared large table concurrently without write conflicts
- How to add a new feature to a large training dataset without waiting 3 to 6 months for logs to accumulate?
- How to let several engineers backfill features into the same training table in parallel without overwriting each other?
Tools and methods
- Two-stage backfill (staging, then promotion) — parallel, low-cost feature attachment with production isolation
- Apache Iceberg — dynamic partition insertion, snapshot rollback, bucketed storage-partitioned joins
- Apache Spark — backfill jobs, including on-demand feature transforms
- Apache Airflow — reusable backfill DAG triggered on demand
- Ray Data — training-time bucket join of staging tables
Google or a one-time email code. No password.