Problem
Canva's product analytics feeds A/B tests, personalization, recommendations and user-facing statistics. It runs at 25 billion events per day with 99.999% uptime. Events are stored indefinitely, so a drifting schema makes years of history unreadable. The first pipeline, built on SQS and SNS, was easy to run, but those two services took about 80% of the cost of the whole service.
Example from the article: a data scientist should be able to read event data created 3 years ago using today's schema versions.
Solution in brief
Every event has a Protobuf schema with full transitive compatibility, two named owners and comments, enforced by a code generator in the repository. Events go through a thin collection server, a Kinesis Data Stream (KDS), an enrichment worker and a separate router. Moving from SQS/SNS to KDS cut cost 20 times. Batched zstd compression and an SQS overflow path cut cost further and keep p99 response time under 20 ms.
Questions this case answers
- How to keep event data readable across schema versions when events are stored indefinitely and producers update at different speeds
- How to cut the transport cost of a high-volume event stream while keeping tail latency of the ingestion endpoint low
- How to make every product analytics event carry a documented schema that stays readable for years?
- How to choose a message bus for a high-volume event pipeline without overspending?
Tools and methods
- Protobuf event schemas — the machine-readable contract for every event
- Datumgen (in-house, built on protoc) — checks compatibility rules and generates code and the event catalog
- Amazon Kinesis Data Streams — transport between server, ingest-worker and router
- Amazon SQS — overflow and failover path for KDS
- zstd batch compression — a few hundred events per record
- Snowpipe Streaming — delivery of events into Snowflake
Google or a one-time email code. No password.