Problem
Pinterest runs thousands of experiments, and by the end of 2024 its experimentation platform computed more than 1,500 user-created metrics daily. Each metric reaches a source table through an ingestion job that the experiment user owns. The metric pipeline waited for all of these jobs, so one late job stalled every metric. Skipped metrics were hard to find and backfill, and the growing load caused out-of-memory (OOM) errors.
Example from the article: a delay in a single upstream ingestion job held back the whole daily metric computation for every experiment.
Solution in brief
The Unified Dynamic Framework (UDF) builds the Airflow DAG at run time from the input data that is actually ready. Ready metrics are processed first in parallel batches, late ones follow in later runs, and a look-back window backfills skipped metrics. A persisted list of metrics in progress prevents double computation, and owners get notified when a delay falls outside the window.
Questions this case answers
- How to compute a large set of user-defined metrics on a daily schedule when each metric depends on an upstream job with its own unpredictable delay
- How to recover metrics skipped by late inputs automatically and tell their owners when recovery is impossible
- How to keep experiment metrics on time when some upstream data jobs finish late?
- How to backfill metrics that were skipped because of upstream delays without double counting?
Tools and methods
- Apache Airflow Dynamic DAG — generates the pipeline structure from input readiness on every run
- Apache Druid — stores computed metrics for the experiment dashboards
- Apache Spark — executes custom metric logic written as Spark functions or queries
- Experiment Metrics Metadata — central store of metric definitions, types and segments
Google or a one-time email code. No password.