Real-time data quality monitoring: Kafka stream contracts with syntactic and semantic test

Yuanzhe Liu26 Nov 2025

Problem

Teams that consume Kafka streams had no automatic way to check the data against agreed rules, so bad records reached downstream systems and stayed unnoticed for a while. Violations are of two kinds. Syntactic: the data does not match the topic schema, for example when a rogue producer uses another schema. Semantic: the schema is valid but a field breaks an agreed pattern, such as the expected length of an identifier. Even after finding a bad record, owners could not tell which field was wrong.

Example from the article: a ticketID field must start with ticketId:, and a record with "ticketID":"123456" violates that rule.

Solution in brief

Stream owners write a data contract: schema, field-level rules, and who to notify. The platform turns each rule into an inverse SQL condition, so the query returns only records that break the contract. A long-running Flink SQL job runs it on the topic and publishes bad records with counts and samples to a dedicated topic, which drives Slack alerts and a UI.

Questions this case answers
  • How to validate a continuous event stream against producer-consumer agreements and flag violating records as they arrive, without affecting existing consumers
  • How to let data owners define many field-level validation rules cheaply when writing them by hand does not scale
  • How to detect bad records in a Kafka topic before downstream consumers break on them?
  • How to let stream owners define field-level validation rules without writing code?

Tools and methods

  • Data contract — schema, field rules and owners in one definition
  • Inverse SQL query — selects only records that violate a rule
  • Apache Flink (Flink SQL) — runs the tests on the stream
  • Apache Kafka — source topic and a topic for results
  • Amazon S3 — archive of quality events
  • Slack — alerts to stream owners
  • LLM-based rule recommendation — suggests field rules from schema and samples
Continue reading — it's free

Google or a one-time email code. No password.