Problem
Walmart's in-house experimentation platform, Expo, computes a fixed set of core operational and transactional metrics for every experiment. That set is not enough when a change touches one narrow interaction and the teams owning other parts of the page want their own numbers. Putting such metrics in the core pipeline needs wide stakeholder agreement and constant code changes. With hundreds of experiments in parallel, a central team cannot build each requested metric.
Example from the article: a homepage redesign moved existing components from above the fold to below it. The teams owning those components wanted to know how customers interacted with them in their new positions.
Solution in brief
Expo UCM (User Custom Metrics) is a self-serve layer that runs next to the core pipeline. A user brings a dataset, defines a metric on it and links the metric to an experiment. Metrics are calculated from SQL templates, and an Airflow job is generated for each source dataset.
Questions this case answers
- How to let many teams define their own experiment metrics on top of existing aggregated datasets while keeping the shared core metric pipeline stable
- How to compute a growing set of per-experiment metrics cheaply so that one failing source dataset does not break the others
- How to let experiment owners add their own metrics to an A/B test without changing the core experimentation pipeline?
- How to measure the effect of a redesign on one specific page component that the core metrics do not cover?
Tools and methods
- A/B test — the experiments whose results the custom metrics extend
- SQL templates — pre-defined query code that the engine modifies per metric definition
- Bring-your-own-dataset (BYOD) — metrics are computed from datasets that domain teams already maintain
- Apache Airflow Dynamic DAGs — one pipeline per onboarded dataset, created automatically
- Apache Spark, Elasticsearch — processing engine and serving store for the dashboard (shown in the architecture figure)
Google or a one-time email code. No password.