Problem
Mercari's search is built on Elasticsearch text matching with a custom score that boosts fresh listings. It ignores what users do: scroll depth, results opened, time spent on an item. Earlier machine-learning ranking attempts showed gains in some A/B experiments but never moved business KPIs significantly. The team wanted a model that learns from behavior and moves sales metrics in the "best-match" search flow.
Example from the article: a click is a weak label because users browse without knowing what they want, so many items are clicked and few are bought.
Solution in brief
Keep Elasticsearch for retrieval and add a second stage: a small neural network, trained with a listwise loss approximating nDCG, that re-ranks the retrieved items. The gain came from labels and data hygiene. Each item gets a graded label from the deepest step of its engagement journey (click, like, comment, purchase), and click-only sessions are dropped. Success is judged on low-variance buyer metrics, not revenue.
Questions this case answers
- How to train a re-ranking model on implicit feedback when single clicks are a noisy, position-biased proxy for relevance and the business goal is purchases
- How to evaluate a ranking change against a business metric when the headline metric has too much variance to detect an effect
- How to improve the order of search results in a C2C marketplace when text matching ignores user behavior?
- How to build training labels for a ranking model when clicks alone are noisy, biased and weakly tied to sales?
Tools and methods
- Elasticsearch — first-stage retrieval (the recall phase) and source of ranking features
- Redis — feature store for daily-updated item click-through rate and impression probability
- Learning-to-rank, listwise loss — a small multi-layer perceptron scores each query-item pair
- Graded relevance labels — label equals the highest-scoring event in the item's engagement journey
- Outlier handling and log1p — keeps long-tailed inputs from destabilizing training
Google or a one-time email code. No password.
