How we scaled ranking with Learn-to-Rank

Zachary Nickerson23 Feb 2026

Problem

Search ranking can learn from clicks and conversions, but only for the 10–20% of queries with high event volume. The other ~80% are unique, low-frequency searches with too little behavior to learn from. These niche queries often carry the highest intent, so a bad order there costs conversions. A team that ships a learned ranker must also explain it and decide which signals feed it.

Example from the article: for the query "wireless dog training collar", which the shop has not seen before, the system has no click history but still knows what a successful ranking looks like from other queries.

Solution in brief

Use the click-based re-ranking that works on head queries as a teacher. Train a listwise learning-to-rank model to predict what that re-ranking would do, from textual relevance plus business signals chosen by the developer. Apply the model to tail queries and inspect signal contributions with SHAP.

Questions this case answers
  • How to extend a ranking policy learned from behavioral feedback to inputs that have too little feedback to learn from
  • How to keep a learned ranker explainable so that its input signals can be tested and pruned
  • How to bring learned ranking to long-tail search queries that have almost no clicks or conversions
  • How to teach a ranking model from the click-based re-ranking you already trust

Tools and methods

  • Dynamic Re-Ranking (DRR) — Algolia's per-query boost of results that users consistently click or buy; the teacher.
  • Listwise learning to rank — optimizes the order of the whole list, in line with metrics such as NDCG.
  • Multi-Signal Ranking (MSR) — the student: a ranker trained on DRR decisions, applied to queries DRR has not seen.
  • SHAP — per-result or averaged contribution of each signal, used to keep or drop signals.
Continue reading — it's free

Google or a one-time email code. No password.