Problem
Teams that consume Kafka streams had no automatic way to check the data against agreed rules, so bad records reached downstream systems and stayed unnoticed for a while. Violations are of two kinds. Syntactic: the data does not match the topic schema, for example when a rogue producer uses another schema. Semantic: the schema is valid but a field breaks an agreed pattern, such as the expected length of an identifier. Even after finding a bad record, owners could not tell which field was wrong.
Example from the article: a ticketID field must start with ticketId:, and a record with "ticketID":"123456" violates that rule.
Solution in brief
Stream owners write a data contract: schema, field-level rules, and who to notify. The platform turns each rule into an inverse SQL condition, so the query returns only records that break the contract. A long-running Flink SQL job runs it on the topic and publishes bad records with counts and samples to a dedicated topic, which drives Slack alerts and a UI.
Questions this case answers
- How to validate a continuous event stream against producer-consumer agreements and flag violating records as they arrive, without affecting existing consumers
- How to let data owners define many field-level validation rules cheaply when writing them by hand does not scale
- How to detect bad records in a Kafka topic before downstream consumers break on them?
- How to let stream owners define field-level validation rules without writing code?
Tools and methods
- Data contract — schema, field rules and owners in one definition
- Inverse SQL query — selects only records that violate a rule
- Apache Flink (Flink SQL) — runs the tests on the stream
- Apache Kafka — source topic and a topic for results
- Amazon S3 — archive of quality events
- Slack — alerts to stream owners
- LLM-based rule recommendation — suggests field rules from schema and samples
Google or a one-time email code. No password.