Multi-dispatch event logging

Charlie Sabine13 Jan 2022

Problem

Deliveroo engineers monitor services with Datadog counters, such as "payment failed" divided by "payment attempted" for a success rate. Tags on these metrics must stay low-cardinality, so a user ID cannot be added. Logs and traces carry detail but are sampled and short-lived. When a bug spikes a metric, the team cannot say how many unique users it hit, whether they retried successfully, or whether first-time users churned afterwards.

Example from the article: a payment bug spikes failures. Did a few users fail many times, or many users once, and what did it cost in revenue and retention?

Solution in brief

Publish every event once through a generic publisher that dispatches to two sinks. A lightweight event (name plus low-cardinality tags) goes to StatsD for alerts, and the full payload with user IDs and correlation IDs goes to a Kafka topic that lands in Snowflake. Impact analysis then runs in the warehouse on those same events.

Questions this case answers
  • How to quantify the user and business impact of system errors when monitoring metrics cannot carry high-cardinality identifiers and sampled logs lose history
  • How to measure how many users and how much revenue a production bug affects, when monitoring only has aggregate counters?
  • How to link operational monitoring events with user-level data in the warehouse?
  • How to set alert thresholds based on business impact instead of a guessed success rate?

Tools and methods

  • Multi-dispatch publisher — one call in application code that emits to both StatsD and Kafka
  • Datadog with StatsD counters — real-time metrics, monitors and alerts
  • Apache Kafka — carries full event payloads with a schema defined upfront
  • Snowflake — warehouse with Kafka topic data and replicated databases, queried ad hoc and through Looker
  • Apache Flink — optional: joins several Kafka topics into streaming business metrics for real-time alerts
Continue reading — it's free

Google or a one-time email code. No password.