Learning to rank diversely

Malay Haldar, Liwei He, Moose Abdool30 Jan 2023

Problem

A ranker that scores each listing by its own booking probability fills every position with the choice most guests made in the past. In Airbnb search this means a page of cheap, similar listings. A minority of guests prefers quality: in Rome, for two guests and two nights, about 20% of bookings carry 50% of booking value, and for them cheaper is not better. These guests are served worse, and the page offers little choice to anyone.

Example from the article: in the Rome two-night, two-guest segment, booked prices follow a log-normal distribution, yet the ranker follows the majority preference at every position, so economical listings crowd out quality ones.

Solution in brief

Train a second neural network that estimates how similar a listing is to the listings already placed above it. Subtract this similarity from the booking-probability score and build the result page greedily, one position at a time. Similar listings drop down, so quality listings enter the top positions.

Questions this case answers
  • How to rebuild a ranked result list so it covers the varied preferences of a population when a pointwise relevance score favors the majority at every position
  • How to learn item-to-item redundancy from interaction logs without similarity labels
  • How to stop a search ranking from filling every position with near-identical listings
  • How to account for similarity between results when the ranker scores each item on its own

Tools and methods

  • Pairwise ranking network — existing model that predicts which of two listings is more likely to be booked.
  • Similarity network — companion model that scores similarity between a candidate and an already placed listing.
  • Greedy sequential re-ranking — fills each position with the best similarity-discounted listing.
  • Online A/B test — measures bookings, booking value and 5-star ratings.
Continue reading — it's free

Google or a one-time email code. No password.