Problem
The Ranking and Auction teams of a hotel-search site test every algorithm change in an A/B test. Such a test needs millions of user visits, which can take several weeks. The wait slows iteration, and a losing variant can cost revenue while it runs. The team wanted a fast, cheap first signal.
Example from the article: testing the hypothesis that users prefer accommodations closer to the city center, with the control list sorted by relevance score and the test list sorted by distance to the center.
Solution in brief
Build a service that replays a random sample of recorded production requests through the ranking service in a separate environment, with the algorithm change switched on by a feature flag. Compare control and flagged responses to see how the list changes and what the revenue impact might be. Run an A/B test only for surviving variants, because replay cannot show clicks.
Questions this case answers
- How to screen candidate changes to a ranking service cheaply, before an online experiment that needs weeks of traffic and risks revenue loss
- How to get early feedback on a ranking change before running a multi-week A/B test?
- How to compare several candidate ranking algorithms without spending live traffic on each?
- How to estimate how a ranking change reshuffles the result list and where revenue might move?
Tools and methods
- Request replay simulation — re-run sampled production requests against the ranking service with and without a change
- Feature flags — switch an algorithm change on per request, so several modifications run independently in one simulation
- Apache Kafka — carries sampled requests, simulation results and progress statistics
- Apache Hadoop — data lake where raw simulation results are stored for analysis
- Elasticsearch — indexes progress statistics for dashboards (the Elastic stack, with Kibana)
Google or a one-time email code. No password.