The Juggler Model: Balancing Expectations in Lodging Rankings

Tiago Cunha19 Sep 2023

Problem

A marketplace ranking serves travelers, properties and the platform at once. Ranking by traveler relevance alone hurts long-term marketplace health, so teams add business adjustments to the relevance score. One fixed weighting is wrong for many searches: with few good options relevance should dominate, with many good options the business objectives can count more. Setting weights per search by hand does not scale, and a Pareto-front search is too slow per request.

Example from the article: three properties with relevance and one adjustment are ranked 3–1–2 (A, B, C) by the plain sum, and 2–3–1 once the search gets weights of 1.3 and 0.3.

Solution in brief

Keep the additive score and give each component a weight shared by all items in a search. A model maps a summary of the search (destination, dates, price histograms) to the best weight set. Labels come from offline simulation: re-rank historical searches with candidate weights and keep the set that scores best on chosen ranking metrics.

Questions this case answers
  • How to combine a relevance score with several business adjustments in one ranking when the best trade-off differs between requests
  • How to choose per-request weights for a linear sum of objectives in real time without searching the Pareto front online
  • How to balance traveler relevance with provider and marketplace objectives in one ranking?
  • How to set the weights of business adjustments per search instead of one global value?

Tools and methods

  • Linear scalarization — one weight per score component, so the ranking stays explainable.
  • Meta-learning (algorithm selection) — picks weights per search, as algorithm selection picks an algorithm per dataset.
  • Offline simulation against a policy — a policy is a set of ranking metrics; it yields labels.
  • Gradient boosting classifier (LightGBM, per the paper) — predicts the weight set from the search summary.
Continue reading — it's free

Google or a one-time email code. No password.