Problem
Numbers in reports disagree, and the cause is often not different metric definitions but broken or inconsistent sources. The larger the system, the harder it is to watch: a microservice fails or someone errs, data drifts, and users never learn about it. The data quality team at AliExpress CIS, a marketplace, built monitoring from zero. Missing or contradictory data costs revenue directly.
Example from the article: two sources that should hold identical product data did not. Part of the catalog never reached the search results, so a shopper looking for toy kittens found nothing and left.
Solution in brief
Check data on every layer of the warehouse: reconcile what was loaded against the source, compare equivalent sources with each other, check business values against a normal interval, and add technical data tests. Reconciling two sources every day recovered the lost products. Split checks into P0 and P1 and keep the logic simple, or alerts pile up.
Questions this case answers
- How to build a data quality monitoring practice from zero across all pipeline layers, with checks matched to the layer where each error is caught
- How to find records that never reach downstream systems because two equivalent sources disagree, by reconciling them against each other
- How to build a data quality monitoring practice from scratch
- How to find products or records that never reach the search index because two sources disagree
Tools and methods
- Reconciliation — everything taken from source A must land in B
- Consistency check — compare equivalent sources, flag mismatches
- Business checks — compare a value for an interval with its normal range
- Data unit tests — duplicates, nulls, field formats
- In-house monitoring and alerting tool
- Soda — being introduced for basic technical tests
- Hadoop, Tableau, Slack — storage of results, dashboards and notifications
Google or a one-time email code. No password.