Powering real-time magic moments through notifications

Ronnie Gilkey, Brian Tombul16 Apr 2024

Problem

Whatnot, a livestream shopping marketplace, uses notifications to bring users back when a show starts, a transaction updates or someone interacts with them. Its notification code grew from a class hierarchy into RabbitMQ background tasks, and logic and volume kept rising. Message behavior was defined through inheritance, mixins and method overrides, so it was hard to understand and debug. Tests mocked internals and never checked rendering or blocking by other components. Teams also wanted analytics and experiments.

Example from the article: a push for an auction that has just started must go out at once, while a large email campaign can wait and must not delay it.

Solution in brief

Whatnot rebuilt notifications as a platform: one type-safe send API, one declarative class per message type, and a pipeline of single-purpose layers that each change a batch of messages. Dispatchers send through Celery queues separated by priority. A dark launch and staged rollouts proved the load before real sends.

Questions this case answers
  • How to build a message-delivery platform where many product teams define messages declaratively and every send passes the same checks for expiry, subscriptions, experiments and rate limits
  • How to migrate a high-volume delivery pipeline to a new architecture and prove it handles the load before real messages go out
  • How to structure a multi-channel notifications platform so product teams can add and A/B test messages safely?
  • How to keep urgent notifications fast when bulk campaigns and a failing provider share the same system?

Tools and methods

  • Declarative message type — templates, priority, TTL, rate limit and data loader in one class
  • Layered pipeline — expiry, channels, subscriptions, experiments, rendering and rate limit as layers
  • Celery queues per priority — urgent sends and providers stay isolated
  • A/B test — an experiment layer picks the template variant per recipient
  • Dark launch and staged rollout — full pipeline, dropped before the provider call
  • RED metrics and SLOs — rates, errors, durations
Continue reading — it's free

Google or a one-time email code. No password.