Multi-dispatch event logging

Charlie Sabine13 Jan 2022

Problem

Deliveroo alerts on StatsD counters in Datadog: "payment attempted" divided into "payment successful" gives a success rate. Counters show that something fails, but not how many unique users or how much revenue a bug touches. A user ID as a tag is unaffordable in a metrics system. Logs and APM traces carry such detail, but they are sampled and short-lived. They locate a bug but cannot size it. Metrics also sit apart from the warehouse, so teams quote different numbers.

Example from the article: a bug spikes payment failures. To judge it, the team must know whether a few users hit many errors or many users hit one, whether they can retry, and whether a new user failing on a first payment churns for good.

Solution in brief

Application code calls one generic publisher that sends each event to two sinks. The full event, with user IDs and correlation IDs, goes to a Kafka topic that lands in Snowflake. A light copy with the event name and low-cardinality tags goes to StatsD for Datadog alerts.

Questions this case answers
  • How to make one instrumentation call serve both real-time alerting and user-level impact analysis when the alerting store cannot hold high-cardinality fields
  • How to quantify who is affected by a production failure and what it costs downstream when operational metrics are only aggregate counts
  • How to log a product event once so that engineers can alert on it and analysts can measure how many users it affected?
  • How to estimate the business impact of a bug, such as affected users and later retention, when monitoring counters cannot carry user IDs?

Tools and methods

  • Multi-dispatch publisher — one call that writes to Kafka and StatsD
  • Datadog, StatsD — real-time counters and alerts
  • Apache Kafka — full event payloads with an upfront schema
  • Snowflake, Looker — impact analysis and dashboards, joined with replicated production tables
  • Apache Flink — optional real-time joins of topics
Continue reading — it's free

Google or a one-time email code. No password.