Problem
A/B tests that hurt the product cost revenue every minute they stay on. Expedia Group runs thousands of tests a year, and its batch system only gives reliable roll-out and stop decisions on long-term data. The first 24 hours of a test had no coverage.
Closing that gap needs live counts of exposed users and metrics per variant. Each user must be counted once, bots must be excluded even when they are flagged late, and the first result must appear within 10 minutes of launch.
Example from the article: an experiment with an unexpected –39% conversion impact was switched off within minutes of launch.
Solution in brief
Expedia built a Flink streaming job, part of its Circuit Breaker, that filters events, keeps per-user state in memory and emits non-bot user counts every 5 minutes. A two-window aggregation then recomputes the full 24-hour totals per experiment and variant, which keeps the counts consistent with the latest bot labels. The statistical decision step is outside this post.
Questions this case answers
- How to compute cumulative per-group statistics from a high-volume event stream with exactly-once counting per user when users can be reclassified retroactively
- How to detect a harmful change in the first minutes of a randomized experiment, before batch analysis is available
- How to detect a harmful A/B test within minutes of launch instead of waiting for the daily batch analysis?
- How to count exposures and metrics once per user in a streaming job when continuous metrics need sums and sums of squares?
Tools and methods
- Apache Flink — stateful stream processing with event-time handling and exactly-once semantics
- Apache Kafka — sources for exposures, clickstream events and bot classifications
- Guardrail monitoring — automatic suspension of a test that shows a negative business outcome in its first 24 hours
- Two-stage (partial, then final) aggregation — spreads work over the whole cluster before grouping by experiment
Google or a one-time email code. No password.