End-to-End Data Infrastructure and Processes for DE/DA/DS

Alexander Tolmachev25 May 2024

Problem

In a large product company, analysts, data engineers and data scientists end up on separate stacks and report to different executives. Analytics blames blocked queries and bad data; engineers blame bad queries. Neighboring teams build almost the same thing in different ways, and a metric computed in BI stops matching the warehouse. The talk comes from Ozon.Fintech (bonuses, cards, installments).

Example from the article: a manager sees one value of a metric in a dashboard and another in the tables, and nobody can reconstruct the analyst's BI formula.

Solution in brief

Give every team the same delivery path, layers and quality rules. Data gets one of three levels (common, key, prod) with requirements that grow with business importance. New data goes through one migrations repository and change data capture, key metrics live on two warehouse layers (DDS and CDM), and DS models run as ordinary services on that data.

Questions this case answers
  • How to give analysts, engineers and data scientists one shared delivery path, quality tiers and metric definitions so that a key metric has one value across dashboards, tables and models
  • How to set data quality requirements in proportion to how much business processes depend on each dataset
  • How to make data analysts, data engineers and data scientists work on one shared data infrastructure and process
  • How to set different quality requirements for data depending on how much the business depends on it

Tools and methods

  • Three levels of data — common, key, prod.
  • Migrations repository and CI/CD — numbered SQL history per table.
  • Change data capture — Debezium, Kafka, then HDFS/S3 or ClickHouse.
  • DDS and CDM layers — detailed and aggregated, with automatic Data Quality.
  • Metric glossary and hierarchy — one owner and one calculation per key metric.
Continue reading — it's free

Google or a one-time email code. No password.