Data Quality as part of the Data Catalog at SIBUR. Experience implementing the practice in a large organization

Alexander Berger25 May 2024

Problem

SIBUR is a petrochemicals company with more than 200 data sources and, according to the speaker, more than 50,000 employees. Its data quality (DQ) checks ran in SAS Data Quality on top of Vertica and were written by a DQ engineer. The company had just started training colleagues to write checks themselves when the vendor left. No commercial tool for their stack was available in Russia, and open-source DQ tooling was thin. The team had to rebuild the practice without a vendor.

Example from the article: a table with ten checks, nine passed and one critical failed, turns red until the check is fixed and re-run.

Solution in brief

SIBUR built its own data quality service as a module of its in-house data catalog. It uses Great Expectations check templates plus custom SQL, and runs them through Kafka against Vertica and source systems. Business-side Data Stewards, paired with data architects, write the checks. Results come back to the catalog as a traffic light per table and attribute.

Questions this case answers
  • How to let domain owners write data quality checks themselves and show their results as a status next to each table in a data catalog
  • How to decide which failed data checks should raise an incident and which only go into statistics, using check weights
  • How to build data quality checks for a warehouse that analysts can extend themselves
  • How to replace a departed data quality vendor with an open-source stack

Tools and methods

  • Great Expectations — check templates; notifications, email and UI are custom
  • Kafka, Docker microservices — check scheduling and parallel execution
  • Vertica, PostgreSQL — checked warehouse; the DQ service's own database
  • Airflow, NiFi, Spark — Data Engineering Stack, triggers checks after loads
  • Data Steward self-service — business owner plus data architect write checks
  • Traffic-light aggregation — check weight sets color and incident
Continue reading — it's free

Google or a one-time email code. No password.