Problem
Every experiment starts as a custom project: an engineer sets up assignments, a data engineer builds metric pipelines, a data scientist runs the statistics. Repeated for each team, this makes experimentation hard to turn into a habit. Metrics drift too: one name means different things in different tables, so a launch or rollback decision depends on which definition someone picked. Non-experts who run analyses themselves can pick a method that does not fit the data and get intervals that are too narrow.
Example from the article: an AI agent answers a PM's question about weekly active users with 1.61M (any event). Growth counts 1.24M (login within 7 days), finance 0.92M (paid action).
Solution in brief
Databricks built the platform on its own internal workspace in four layers: certified metrics in a central registry, assignments derived from raw exposure events behind one table function, analyses as typed library classes run by a harness from a YAML spec, and a thin app on top. Hundreds of changes are tested monthly.
Questions this case answers
- How to run self-serve controlled experiments where assignments are derived from raw exposure events and metric definitions are governed centrally
- How to let non-statisticians run analyses safely by restricting them to methods that fit the data
- How to turn one-off, manually run experiments into a self-serve experimentation platform
- How to make experiment metrics consistent when several teams define the same metric differently
Tools and methods
- Unity Catalog Metric Views and Metric Store — metrics defined in YAML, certified and shared
- Exposure events +
assignmentstable function — one API for user-level assignment history - Analysis class library and harness — typed analyses, results stored in a data model
- YAML experiment spec — plain-text interface for people and agents
- Lakeflow Serverless Jobs, Lakehouse Apps — job runs and UI
Google or a one-time email code. No password.