Problem
Expedia Group's lodging ranking has three stages: selection, relevance ranking and re-ranking with business adjustments. Every change to the stack is A/B tested on conversion rate (CVR). Lodging bookings are rare, so a small CVR uplift needs a huge sample to detect. Tests can run for months and are exposed to sample pollution, user adaptation and seasonality. This limits how many ranking ideas the team can try.
Example from the article: with a base CVR of 3%, a relative uplift of 0.3%, 80% power and a 10% significance level, a two-sided test of proportions needs about 92 million users across both variants.
Solution in brief
Interleaving blends the results of two rankers into one list. Every user sees both variants and acts as their own control. Clicks and bookings are attributed back to the ranker that produced the item, and the ranker that wins more searches (or users) is preferred. Expedia uses it to prune candidates before A/B testing.
Questions this case answers
- How to compare two ranking variants with far fewer users than a between-group A/B test needs when the target conversion event is rare
- How to screen many candidate rankers cheaply and send only the best ones to a slow, expensive randomized test
- How to find the winning ranking models before committing months to an A/B test?
- How to compare two rankers when conversion is too rare for a classic A/B test to reach significance?
Tools and methods
- Interleaving test — within-subject comparison of two rankers on a single blended list
- Backtesting — offline pre-filter of candidate rankers on NDCG or revenue-focused metrics
- Win/tie/loss lift metric — direction of user preference, computed per search or per user
- t-test on win indicators — fast confidence interval, close to the bootstrap result
- A/B test — final confirmation of the business effect
Google or a one-time email code. No password.