Building a User Signals Platform at Airbnb

Kidai Kwon, Pavan Tambay, Xinrui Hua, Soumyadip (Soumo) Banerjee, Phanindra (Phani) Ganti20 Nov 2024

Problem

Personalization at Airbnb needs to know what a guest just did (searches, viewed homes, wishlists) and which stage of the trip the guest is in. The data must be available for online serving in under 1 second and for offline analysis. Most product teams do not know stream processing. Failures in the streaming layer also create event backlogs that make signals stale.

Example from the article: an "active trip planner" segment puts a guest in as soon as they search and removes them after 14 days of inactivity or on a booking.

Solution in brief

A User Signals Platform (USP) reads Kafka events with Flink and writes versioned records to a key-value store. Developers add signals, segments and session summaries through a config plus a small transform class, and scripts generate the Flink, backfill and alert files. Hot-standby workers keep jobs from stalling on failures.

Questions this case answers
  • How to turn raw product events into near real-time per-user signals, segments and short-term engagement summaries that many teams can define without stream-processing expertise
  • How to keep a partitioned stream-processing job available when a single worker fails
  • How to make recent user actions (searches, viewed listings, bookings) queryable within a second for personalization?
  • How to maintain user segments such as active trip planners with start and expiry rules in near real time?

Tools and methods

  • Apache Flink — event-by-event stream processing of Kafka events, stateful joins with RocksDB, windowed aggregation
  • Apache Kafka — source events and transformed signal events that trigger downstream jobs
  • Key-value store (append-only) — storage of signals and segments, queried by signal type and time range
  • Lambda architecture — online stream path plus an offline path for correction and backfill
  • User segments — near real-time cohorts defined by trigger, start and expiration rules
  • Session windows and sliding windows — short-term engagement aggregation per user
Continue reading — it's free

Google or a one-time email code. No password.