D3: An Automated System to Detect Data Drifts

Anshal Shukla, Kousik Nath, Nipun Vats, Vineeth Tatipathri, Dinesh Jagannathan23 Feb 2023

Problem

Uber's ML models and leadership dashboards depend on 1000+ tier 1 datasets. The most damaging data incidents are partial and silent: some rows or column values go missing, no pipeline fails, and no headline metric moves visibly. Users find them weeks or months later, after bad decisions and model training on corrupted data. Partial-data incidents took on average 5X longer to detect than incidents caused by ETL or infrastructure failures. Hand-written static-threshold tests do not scale to petabytes, shifting trends and hundreds of cities.

Example from the article: a fare component was missing in the fares dataset for 10% of sessions across key US cities after an app experiment changed fare logging. A data scientist found it manually after 45 days.

Solution in brief

D3 (Dataset Drift Detector) computes daily column statistics ("monitors"), also per city or app version, and forecasts each one's expected range from 90 days of history with Prophet. A value outside conservative limits raises an alert. Onboarding is automatic: columns are chosen by usage, monitors by data type.

Questions this case answers
  • How to detect silent partial data loss across a very large number of tables without per-table manual rules
  • How to set alert thresholds automatically for many daily data-quality metrics while keeping false alerts low
  • How to catch silent partial-data incidents in thousands of tables before users notice them weeks later
  • How to monitor column-level drift without hand-written static thresholds

Tools and methods

  • Column monitors — null, false and zero percentage, percentiles, mean, standard deviation, distinct count, as SQL Jinja templates
  • Prophet — forecasts expected limits for each monitor's daily time series
  • Apache Spark — generic jobs that compute monitor stats with batched queries
  • Apache Hive — table that stores stats and dynamic thresholds
  • Tableau — dashboards of monitor trends and alerts
Continue reading — it's free

Google or a one-time email code. No password.