Metrics Board: Building an Agent-ready Metrics Layer

Michele Ceccacci, Jason Coffman, Colm O'Shaughnessy, Laura Palmer, Adam Podraza, Surya Karri6 Oct 2026

Problem

Hundreds of analysts, data scientists and engineers at Pinterest create thousands of metrics. Each team built its own pipeline and wired it by hand into reporting and experimentation tools. Building similar pipelines was slow, monitoring was inconsistent so breakages surfaced only downstream, people did not know existing metrics and created duplicates, and metadata drifted between systems with no source of truth. Thousands of experiments gate feature launches, so metrics that are untrusted or hard to add slow launches. Agents add a demand: metrics must be discoverable, quality-checked and machine-readable.

Example from the article: a project revamping key engagement metrics saved roughly two engineering-months compared with the old approach.

Solution in brief

Metrics Board is a thin semantic layer. Every metric is a YAML spec in a Git repo, and that single definition generates the compute pipeline, validations, catalog entry, experiment availability and BI datasets. Trust is shown through metric grades and dashboard tiers, and an MCP server gives agents the same metadata.

Questions this case answers
  • How to keep thousands of metrics consistent, quality-checked and discoverable when many independent teams define them over heterogeneous tables
  • How to expose governed metric definitions to LLM agents so answers rest on certified definitions rather than ad hoc SQL
  • How to define product metrics once so that pipelines, quality checks, catalog entries and experiment metrics are generated from the definition?
  • How to stop metric duplication and metadata drift when hundreds of people create metrics?

Tools and methods

  • YAML metric spec in Git — single source of truth with review and history
  • Apache Airflow — one generated DAG per metric
  • SQLGlot — parses metric SQL to derive upstream dependency waits
  • Rule-based checks and ML anomaly detection — validations declared in the spec
  • DataHub (PinCat), Superset, StarRocks, Helium — publishing targets
  • MCP server — lets LLM tools find and read metric definitions
Continue reading — it's free

Google or a one-time email code. No password.