Problem
VWO (Wingify's A/B testing product) shows customers aggregates of their visitors' events on a dashboard, and customers define the metrics themselves. A metric is an aggregate function over events that meet customer-set conditions, for example visitors who are female and from India, or who placed an order of at least $499. The dashboard must answer in under a second. Computing this from raw events at read time made the analytical database use more CPU as data and concurrent queries grew.
Example from the article: for such conditions, show the number of matching events, unique visitors, total item value and total item quantity.
Solution in brief
Compute metrics while events are written. A streaming pipeline evaluates each customer's conditions on every incoming event and increments day-wise counters in a key-value store. The dashboard reads ready numbers instead of scanning raw events. The catch: a condition may depend on a single event only.
Questions this case answers
- How to serve user-defined conditional aggregates with sub-second latency when computing them from raw events at read time becomes too expensive
- How to maintain per-entity and unique-entity counters in a stream when metric definitions are not known in advance and the table schema cannot be fixed
- How to let customers define their own conditional metrics and still show them on a dashboard in under a second
- How to count unique visitors per metric in a streaming pipeline
Tools and methods
- Apache Beam — streaming job that computes metrics from incoming events
- Google Bigtable — day-wise metric counters; built-in increment operator
- Redis — side input with each customer's metric definitions
- ClickHouse — raw events; first tried for serving metrics, still the fallback for conditions the stream cannot evaluate
- BigQuery — stores raw events; too slow for the sub-second target
- BlazeMeter — load tests comparing the two designs
Google or a one-time email code. No password.