Computing Aggregates in VWO

Anil Mor9 Mar 2023

Problem

VWO, an A/B testing product, shows customers a dashboard of metrics built from their visitors' events. Customers define both the conditions and the metrics, for example: count only visitors who are female and from India, or who placed an order worth at least $499, then report the number of such interactions, unique visitors, items sold and sales value. Running such queries over raw events at read time was too slow in BigQuery (seconds) and too CPU-hungry in ClickHouse as data and parallel queries grew. The dashboard had to answer in under a second on limited hardware.

Example from the article: a two-step e-commerce funnel (visitors reaching the product page, visitors who buy) tracked day by day, where each customer adds arbitrary conditions on top.

Solution in brief

Compute the metrics once, while events are being ingested, and store only the running counters. A streaming job evaluates each customer's conditions on every event and increments day-wise counters in Bigtable; the dashboard reads those counters directly. Metric definitions come to the job from Redis.

Questions this case answers
  • How to serve user-defined conditional aggregates with sub-second latency when scanning raw events at query time exhausts CPU
  • How to maintain an open-ended set of counters from an event stream when a fixed relational schema cannot hold them
  • How to let customers define their own conditional metrics and still show them on a dashboard in under a second?
  • How to compute metrics at event ingestion time instead of scanning raw events at query time?

Tools and methods

  • Bigtable — stores day-wise metric counters; atomic increments from many threads
  • Apache Beam — streaming write service that evaluates conditions and computes metrics
  • Redis — in-memory side input with client metric definitions for the pipeline
  • ClickHouse, BigQuery — raw event storage; kept for ad hoc and cross-event queries
Continue reading — it's free

Google or a one-time email code. No password.