Problem
Search ranking can learn from clicks and conversions, but only for the 10–20% of queries with high event volume. The other ~80% are unique, low-frequency searches with too little behavior to learn from. These niche queries often carry the highest intent, so a bad order there costs conversions. A team that ships a learned ranker must also explain it and decide which signals feed it.
Example from the article: for the query "wireless dog training collar", which the shop has not seen before, the system has no click history but still knows what a successful ranking looks like from other queries.
Solution in brief
Use the click-based re-ranking that works on head queries as a teacher. Train a listwise learning-to-rank model to predict what that re-ranking would do, from textual relevance plus business signals chosen by the developer. Apply the model to tail queries and inspect signal contributions with SHAP.
Questions this case answers
- How to extend a ranking policy learned from behavioral feedback to inputs that have too little feedback to learn from
- How to keep a learned ranker explainable so that its input signals can be tested and pruned
- How to bring learned ranking to long-tail search queries that have almost no clicks or conversions
- How to teach a ranking model from the click-based re-ranking you already trust
Tools and methods
- Dynamic Re-Ranking (DRR) — Algolia's per-query boost of results that users consistently click or buy; the teacher.
- Listwise learning to rank — optimizes the order of the whole list, in line with metrics such as NDCG.
- Multi-Signal Ranking (MSR) — the student: a ranker trained on DRR decisions, applied to queries DRR has not seen.
- SHAP — per-result or averaged contribution of each signal, used to keep or drop signals.
Google or a one-time email code. No password.