Experience Building a Data Quality Tool at MTS

Andrey Azeev25 May 2024

Problem

MTS runs a shared data platform: 500+ data products, 1000+ source systems, 20+ petabytes. Teams needed one place to define quality checks, see results and history, and get alerts, without storing sensitive data. After the first version launched, teams started wrapping every storage object in checks. Check queries began competing for storage compute with the Data Lake's own scheduled jobs, the DQ host ran short of capacity, and alerts arrived late.

Example from the article: each set of checks started its own Spark Driver in client mode on the DQ host, so more checks meant longer runs and delayed alerts.

Solution in brief

Cap and split resources by team, then cut the load itself: run workers inside each product's perimeter, build trend checks on already stored DQ results instead of the source, and compute metrics inside ETL through a Python SDK. A profiler helps pick checks that matter; extra services add failing-record samples and cross-database checks.

Questions this case answers
  • How to run data quality checks on a shared large storage without querying the source more than once and without starving other scheduled jobs
  • How to pick data quality checks that catch real problems without producing many false alerts, across fragmented sources that must not store sensitive data
  • How to scale data quality checks in a shared data lake without overloading compute
  • How to compute data quality metrics inside an ETL pipeline without re-reading the source

Tools and methods

  • YAML checks in Git, DQ API — self-service definition and versioning
  • SQL code generation — typical checks merged into one query, source returns aggregates only
  • Hybrid workers (RabbitMQ, scheduler) — shared workers plus workers inside a product's perimeter
  • Python SDK with sampler — metrics over a cached Spark DataFrame, results sent to the API
  • Data profiler (Python library, JSON in S3, UI) — anomaly rules and rule creation from the UI
  • Trino — cross-database SQL checks
Continue reading — it's free

Google or a one-time email code. No password.