Problem
Deliveroo engineers monitor services with Datadog counters, such as "payment failed" divided by "payment attempted" for a success rate. Tags on these metrics must stay low-cardinality, so a user ID cannot be added. Logs and traces carry detail but are sampled and short-lived. When a bug spikes a metric, the team cannot say how many unique users it hit, whether they retried successfully, or whether first-time users churned afterwards.
Example from the article: a payment bug spikes failures. Did a few users fail many times, or many users once, and what did it cost in revenue and retention?
Solution in brief
Publish every event once through a generic publisher that dispatches to two sinks. A lightweight event (name plus low-cardinality tags) goes to StatsD for alerts, and the full payload with user IDs and correlation IDs goes to a Kafka topic that lands in Snowflake. Impact analysis then runs in the warehouse on those same events.
Questions this case answers
- How to quantify the user and business impact of system errors when monitoring metrics cannot carry high-cardinality identifiers and sampled logs lose history
- How to measure how many users and how much revenue a production bug affects, when monitoring only has aggregate counters?
- How to link operational monitoring events with user-level data in the warehouse?
- How to set alert thresholds based on business impact instead of a guessed success rate?
Tools and methods
- Multi-dispatch publisher — one call in application code that emits to both StatsD and Kafka
- Datadog with StatsD counters — real-time metrics, monitors and alerts
- Apache Kafka — carries full event payloads with a schema defined upfront
- Snowflake — warehouse with Kafka topic data and replicated databases, queried ad hoc and through Looker
- Apache Flink — optional: joins several Kafka topics into streaming business metrics for real-time alerts
Google or a one-time email code. No password.