Problem
Airbnb's relevance team has more search ranking ideas than its A/B tests can handle. Guests book less often than users of other e-commerce sites, and the business metrics are conversion-focused, so each A/B test needs a lot of traffic and time. The team needed an online test that is much faster and still agrees with the A/B result.
Example from the article: a ranker that picks a random listing from the top 300 results and puts it in the top slot. Interleaving reached the same conclusion as the A/B test using 0.5% of the A/B running time and 4% of its traffic.
Solution in brief
Show each user one list that mixes control and treatment results by team drafting. Differing listings form pairs, each a direct duel between the rankers, and bookings won in those pairs decide the winner. It runs on a small slice of traffic and screens ideas before the A/B test.
Questions this case answers
- How to compare two ranking models online with far less traffic and time than a user-split A/B test needs
- How to credit a delayed, rare outcome to one of two systems when the user interacts with both many times before converting
- How to test many ranking ideas online when A/B test traffic is too small for all of them?
- How to compare two rankers online without splitting users into groups?
Tools and methods
- Interleaving (team drafting) — blends control and treatment results into one list
- Competitive pairs — the two differing listings picked at one step; the unit of comparison
- Booking attribution rules — give credit to a team when the booked listing was clicked several times
- 1-sample t-test — decides the winner from the preference margin
- NDCG — offline filter before online tests
- A/B test — final check for rankers that pass interleaving
Google or a one-time email code. No password.