The Data Infra Behind Zillow's 3x Growth in Experiment Volume

Aaron Wroblewski27 Mar 2023

Problem

Zillow teams ran experiments with nearly a dozen third-party or home-built tools, each solving one piece of the stack. Product managers waited weeks for data scientists to build scorecards, metrics, tests and charts. Bought tools measured only metrics local to the tested screen, so goals such as touring a home or completing a move, and effects across lines of business, stayed invisible. Self-service setups did not scale beyond one product area or brand, so experiments were costly and often untrustworthy.

Example from the article: a team had a working assignment service, but product managers still waited weeks for a data scientist to build the experiment scorecard.

Solution in brief

Zillow built one experimentation platform (ZEXP) on a hybrid plan: buy assignment and feature flags as SDKs, build analysis in-house. Three own parts do the work: an analysis library, an API that stores each experiment's configuration, results and status, and a self-service web app, all on shared compute and streaming platforms. Experiment volume tripled in two years.

Questions this case answers
  • How to scale the number of online experiments across many products when every team uses its own assignment, metric and reporting tools
  • How to build an automated, debuggable analysis pipeline whose intermediate data can be inspected and reprocessed selectively
  • How to replace a dozen disconnected experimentation tools with one self-service platform?
  • How to automate experiment analysis so product managers do not wait for data scientists to build scorecards?

Tools and methods

  • ZEXPy — Python/PySpark analysis library with checkpoints per stage.
  • FastAB — API that is the single source of truth for experiments.
  • AB Testing Hub — web app for setup, health alerts, results and decisions.
  • Bought assignment service — SDKs, feature flags, hourly exposure logs.
  • Spark on Kubernetes, Kafka, Flink, Terraform — compute and event pipelines.
  • Data contracts — scheduled data tests with alerts.
Continue reading — it's free

Google or a one-time email code. No password.