Learning-To-Rank: Sorting Search Results

Ilia Ivanov22 Jun 2022

Problem

OLX is a classifieds marketplace with millions of ads. Until about a year before the post, users could sort results only by ad age or price, so they spent a long time comparing ads or adding filters. A hand-built "Recommended" formula over text similarity, ad statistics (CTR) and ad age improved purchases in A/B tests, but a fixed formula cannot learn from logged behavior. The team needed learned weights and a way to compare candidate rankers without an A/B test for each.

Example from the article: a search finds 3 ads and the user buys the second. That session becomes three labeled <query, ad> rows.

Solution in brief

Treat ranking as supervised learning. Each <query, ad> pair gets a binary conversion label and a feature vector. A LambdaMART model (gradient-boosted trees, XGBoost) is trained to optimize NDCG, then uploaded to Solr through its LTR plugin. Candidate models are compared by offline NDCG on historical sessions, and the final one is checked in an A/B test on first-page conversions.

Questions this case answers
  • How to rank candidate items for a query by predicted conversion when the true conversion probability is unobserved and only binary outcomes are logged
  • How to select between ranking models offline so that the offline ordering predicts the online experiment result
  • How to order search results so that the ads most likely to convert appear first?
  • How to replace a hand-tuned ranking formula with a learned ranking model?

Tools and methods

  • Learning to rank (LambdaMART) — trains a tree ensemble to order ads per query by conversion probability
  • XGBoost — implementation of LambdaMART used for training
  • NDCG — offline ranking metric computed on binary purchase labels
  • Solr LTR plugin — serves the trained model inside the search engine
  • A/B test — final check on first-page conversions
Continue reading — it's free

Google or a one-time email code. No password.