Problem
Uber's ML models and leadership dashboards depend on 1000+ tier 1 datasets. The most damaging data incidents are partial and silent: some rows or column values go missing, no pipeline fails, and no headline metric moves visibly. Users find them weeks or months later, after bad decisions and model training on corrupted data. Partial-data incidents took on average 5X longer to detect than incidents caused by ETL or infrastructure failures. Hand-written static-threshold tests do not scale to petabytes, shifting trends and hundreds of cities.
Example from the article: a fare component was missing in the fares dataset for 10% of sessions across key US cities after an app experiment changed fare logging. A data scientist found it manually after 45 days.
Solution in brief
D3 (Dataset Drift Detector) computes daily column statistics ("monitors"), also per city or app version, and forecasts each one's expected range from 90 days of history with Prophet. A value outside conservative limits raises an alert. Onboarding is automatic: columns are chosen by usage, monitors by data type.
Questions this case answers
- How to detect silent partial data loss across a very large number of tables without per-table manual rules
- How to set alert thresholds automatically for many daily data-quality metrics while keeping false alerts low
- How to catch silent partial-data incidents in thousands of tables before users notice them weeks later
- How to monitor column-level drift without hand-written static thresholds
Tools and methods
- Column monitors — null, false and zero percentage, percentiles, mean, standard deviation, distinct count, as SQL Jinja templates
- Prophet — forecasts expected limits for each monitor's daily time series
- Apache Spark — generic jobs that compute monitor stats with batched queries
- Apache Hive — table that stores stats and dynamic thresholds
- Tableau — dashboards of monitor trends and alerts
Google or a one-time email code. No password.