Databricks on Databricks: Build Your Own Experimentation Platform

Donghan Zhang, Nicholas Stanisha18 Jun 2026

Problem

Every experiment starts as a custom project: an engineer sets up assignments, a data engineer builds metric pipelines, a data scientist runs the statistics. Repeated for each team, this makes experimentation hard to turn into a habit. Metrics drift too: one name means different things in different tables, so a launch or rollback decision depends on which definition someone picked. Non-experts who run analyses themselves can pick a method that does not fit the data and get intervals that are too narrow.

Example from the article: an AI agent answers a PM's question about weekly active users with 1.61M (any event). Growth counts 1.24M (login within 7 days), finance 0.92M (paid action).

Solution in brief

Databricks built the platform on its own internal workspace in four layers: certified metrics in a central registry, assignments derived from raw exposure events behind one table function, analyses as typed library classes run by a harness from a YAML spec, and a thin app on top. Hundreds of changes are tested monthly.

Questions this case answers
  • How to run self-serve controlled experiments where assignments are derived from raw exposure events and metric definitions are governed centrally
  • How to let non-statisticians run analyses safely by restricting them to methods that fit the data
  • How to turn one-off, manually run experiments into a self-serve experimentation platform
  • How to make experiment metrics consistent when several teams define the same metric differently

Tools and methods

  • Unity Catalog Metric Views and Metric Store — metrics defined in YAML, certified and shared
  • Exposure events + assignments table function — one API for user-level assignment history
  • Analysis class library and harness — typed analyses, results stored in a data model
  • YAML experiment spec — plain-text interface for people and agents
  • Lakeflow Serverless Jobs, Lakehouse Apps — job runs and UI
Continue reading — it's free

Google or a one-time email code. No password.