The Journey to Machine-Learned Re-ranking

Alexander Zagniotov, Richard Calland1 Jan 2023

Problem

Mercari's search is built on Elasticsearch text matching with a custom score that boosts fresh listings. It ignores what users do: scroll depth, results opened, time spent on an item. Earlier machine-learning ranking attempts showed gains in some A/B experiments but never moved business KPIs significantly. The team wanted a model that learns from behavior and moves sales metrics in the "best-match" search flow.

Example from the article: a click is a weak label because users browse without knowing what they want, so many items are clicked and few are bought.

Solution in brief

Keep Elasticsearch for retrieval and add a second stage: a small neural network, trained with a listwise loss approximating nDCG, that re-ranks the retrieved items. The gain came from labels and data hygiene. Each item gets a graded label from the deepest step of its engagement journey (click, like, comment, purchase), and click-only sessions are dropped. Success is judged on low-variance buyer metrics, not revenue.

Questions this case answers
  • How to train a re-ranking model on implicit feedback when single clicks are a noisy, position-biased proxy for relevance and the business goal is purchases
  • How to evaluate a ranking change against a business metric when the headline metric has too much variance to detect an effect
  • How to improve the order of search results in a C2C marketplace when text matching ignores user behavior?
  • How to build training labels for a ranking model when clicks alone are noisy, biased and weakly tied to sales?

Tools and methods

  • Elasticsearch — first-stage retrieval (the recall phase) and source of ranking features
  • Redis — feature store for daily-updated item click-through rate and impression probability
  • Learning-to-rank, listwise loss — a small multi-layer perceptron scores each query-item pair
  • Graded relevance labels — label equals the highest-scoring event in the item's engagement journey
  • Outlier handling and log1p — keeps long-tailed inputs from destabilizing training
Continue reading — it's free

Google or a one-time email code. No password.