Using Metrics Layer to Standardize and Scale Experimentation at DoorDash

Arun Balasubramani12 Apr 2023

Problem

DoorDash's experimentation platform, Curie, first let data scientists check in their own Snowflake SQL for each experiment's metrics. Teams defined the same business metric differently, and only SQL-literate people could read results. Monolithic queries re-computed the same metrics for every experiment, so full-table scans and joins blocked the worker queue. CUPED, pre-experiment bias checks and slicing by dimensions could not be built on free-form SQL. Metric changes had no owner and no review.

Example from the article: an orders metric should count only deliveries made after a user's first exposure to the experiment, and be sliced by dimensions such as country or platform.

Solution in brief

Metrics are declared once as YAML models (sources with measures and dimensions, then metrics) in Git, validated in CI and served by a central service. A generated pipeline pre-computes raw measures daily. Per experiment it joins them to exposures in Snowflake and returns only moment aggregates for the stats engine.

Questions this case answers
  • How to define experiment metrics once, under review and ownership, and reuse them across many concurrent tests without per-test SQL
  • How to compute metrics for many experiments cheaply when ad-hoc queries repeat the same scans and joins
  • How to keep one agreed definition of each metric across all A/B tests in a company?
  • How to let product managers read experiment results without writing SQL?

Tools and methods

  • Metrics layer (semantic layer) — central repository of source, measure, dimension and metric definitions with owners.
  • Metric packs — team-owned collections of metrics reused across experiments; core packs are added automatically.
  • Snowflake — warehouse for measures and aggregates.
  • Dagster — orchestration; jobs generated from the models.
  • Delta method — inference from moment aggregates computed in the warehouse.
  • CUPED / CUPAC — variance reduction from pre-experiment values or ML predictions as covariates.
Continue reading — it's free

Google or a one-time email code. No password.