Problem
A ranking model for an online store is trained on clicks and purchases, and users click mostly on the top of the list. Positions 1–2 collect far more positive labels than positions 30–40, so the model scores items high because they were shown high, not because they are relevant. Ranking by that score repeats the old ranking, and fresh data from it keeps the loop closed.
Example from the article: search results for "bread" in a grocery store. Most users do not scroll through 30–50 positions and pick something in the top few.
Solution in brief
Separate the position effect from relevance, then remove it from training. Constructor.io measures pure position bias with a shuffle test of the top-6 positions and reweights labels by it. A newer approach adds biased features such as position to a gradient boosting ranker and fakes them at inference.
Questions this case answers
- How to remove position bias from implicit feedback when items shown higher receive more interactions regardless of relevance
- How to measure the pure effect of display position on interaction rates when the current ranking itself decides exposure
- How to correct position bias in a product ranking model trained on click feedback
- How to separate the position effect from relevance in CTR by position
Tools and methods
- Shuffle test — A/B test that shuffles the top-N items to get a CTR-by-position curve without relevance.
- Inverse position weighting — label weights by position, for pointwise or pairwise training.
- Position-aware learning — a bias-carrying feature (position) in the model, neutralized at inference.
- Unbiased subsample — one random item per list moved to a random position, as a clean training set.
- Listwise ranking — judging a whole ranking: bandits, contextual bandits or a network that outputs a list.
Google or a one-time email code. No password.