Building a strong foundation to accelerate StreetEasy’s data science efforts

Dario Marrocchelli22 May 2024

Problem

StreetEasy (Zillow) logged user actions into one event table with billions of rows and hundreds of columns. Even a basic KPI required the analyst to know how each action is encoded. Analysts wrote different SQL for the same KPI and got different values. Simple A/B test reads took hours or days, and few people outside the Data Science team could run an analysis.

Example from the article: the submit rate (share of sessions with at least one request to schedule a showing) needs the schedule_showing event, only the values message_sent and message_sent_app (desktop versus iOS), and the exclusion of message_aborted.

Solution in brief

Transform raw events once, with dbt, into certified session-level datasets whose metric logic is defined, reviewed and tested centrally. Build the A/B test reads, ML prototypes and dashboards on these datasets. Serve the roughly 80% of frequent questions with certified dashboards and leave the rest to fast ad-hoc analysis.

Questions this case answers
  • How to make the same metric come out identical for every analyst when it is computed from raw event logs with platform-specific quirks
  • How to let non-technical stakeholders answer frequent product questions without a data scientist while keeping ad-hoc analysis for the rest
  • How to make product metrics such as submit rate reproducible when every analyst writes their own SQL over raw event logs?
  • How to cut the turnaround of A/B test reads from days to minutes?

Tools and methods

  • dbt — runs the raw-to-certified transformations on a schedule and holds field descriptions.
  • Certified datasets — session-level tables with agreed metric definitions and automated tests.
  • Session table with experiment and variant — makes an A/B test read a single group-by.
  • Certified dashboards and 80-20 analytics — cover frequent and exploratory questions.
Continue reading — it's free

Google or a one-time email code. No password.