Problem
StreetEasy (Zillow) logged user actions into one event table with billions of rows and hundreds of columns. Even a basic KPI required the analyst to know how each action is encoded. Analysts wrote different SQL for the same KPI and got different values. Simple A/B test reads took hours or days, and few people outside the Data Science team could run an analysis.
Example from the article: the submit rate (share of sessions with at least one request to schedule a showing) needs the schedule_showing event, only the values message_sent and message_sent_app (desktop versus iOS), and the exclusion of message_aborted.
Solution in brief
Transform raw events once, with dbt, into certified session-level datasets whose metric logic is defined, reviewed and tested centrally. Build the A/B test reads, ML prototypes and dashboards on these datasets. Serve the roughly 80% of frequent questions with certified dashboards and leave the rest to fast ad-hoc analysis.
Questions this case answers
- How to make the same metric come out identical for every analyst when it is computed from raw event logs with platform-specific quirks
- How to let non-technical stakeholders answer frequent product questions without a data scientist while keeping ad-hoc analysis for the rest
- How to make product metrics such as submit rate reproducible when every analyst writes their own SQL over raw event logs?
- How to cut the turnaround of A/B test reads from days to minutes?
Tools and methods
- dbt — runs the raw-to-certified transformations on a schedule and holds field descriptions.
- Certified datasets — session-level tables with agreed metric definitions and automated tests.
- Session table with experiment and variant — makes an A/B test read a single group-by.
- Certified dashboards and 80-20 analytics — cover frequent and exploratory questions.
Google or a one-time email code. No password.