Problem
A marketplace search shows a list of ads for a query, and someone has to decide their order. A relevance label on an ad means little alone: an iPhone ad and a car ad can both be "relevant" in training data, yet only one fits a given query. Relevance also depends on neighbors, because an ad that looks weak next to a better one may look fine alone. A plain classifier on ad features ignores both the query and the comparison, and does not care where the relevant ads land in the list.
Example from the article: two ads, an iPhone and a car, both labeled 1, but each is relevant only to the person searching for it.
Solution in brief
Group the training data by session, train a pairwise ranker in XGBoost with "objective": "rank:pairwise" (RankNet), and score the result with NDCG. To weight the top of the list, switch to "objective": "rank:ndcg" (LambdaRank-style), which scales each pair's gradient by the NDCG change that swapping the pair would cause.
Questions this case answers
- How to train a ranker for a list of items shown together when relevance only makes sense relative to the other items in the same list
- How to optimize a ranking model for a top-weighted quality metric when the metric itself is not differentiable
- How to train a model that orders ads inside one search listing instead of classifying each ad as relevant or not?
- How to build a training table for a ranker from sessions and user clicks?
Tools and methods
- XGBoost (XGBRanker) — gradient-boosted trees with ranking objectives
- RankNet (
rank:pairwise) — minimizes the number of wrongly ordered ad pairs per session - LambdaRank (
rank:ndcg) — pairwise gradients weighted by the NDCG change - NDCG@k — ranking quality metric, averaged over listings
- scikit-learn
ndcg_score— computes NDCG per listing
Google or a one-time email code. No password.