Event Tracking System at Wealthfront

Gurmeet Saran7 Nov 2022

Problem

Wealthfront, a robo-advisor, used event data from web, iOS and Android for marketing, A/B testing and retention analysis. The same event looked different on each platform. Definitions lived inside the code repositories of the teams that logged them, so data science and product management had to write custom downstream logic to reconcile them. The old system also tolerated occasional data loss, which new models could not accept, and it had to scale higher.

Example from the article: an event sent from iOS, Android and Web differed in shape, and analysts patched the differences in each downstream pipeline.

Solution in brief

Define every event once in a typed schema (Avro) and generate client code and warehouse table definitions from it. A registry checks each schema change for compatibility before it ships. Send events through a managed queue-and-batch pipeline with retries, a dead letter queue and automatic alerting.

Questions this case answers
  • How to keep event definitions consistent across several client platforms when each team hard-codes them in its own repository
  • How to change an event schema over time without breaking consumers when old client versions and delayed events keep arriving
  • How to make the same product event look identical when it is logged from web, iOS and Android?
  • How to keep one source of truth for event definitions that analysts and engineers can read?

Tools and methods

  • Avro event schema — one definition per event; @glue syncs it to the table catalog, @swagger generates client code
  • AWS Glue Schema Registry (FULL_ALL mode) — rejects schema changes that break reading of data written under any earlier version
  • Swagger Codegen — builds web and mobile client code from the schema
  • API Gateway, Kinesis Data Streams, Kinesis Firehose, S3 with Parquet, Athena — ingestion, storage and querying
  • AWS Lambda, SQS, dead letter queue, CloudWatch, EventBridge, PagerDuty — retries, failure isolation and alerting
Continue reading — it's free

Google or a one-time email code. No password.