Inverse Propensity Score based offline estimator for deterministic ranking lists using position bias

Sumit Sidana, Nick Wood26 Aug 2022

Problem

Teams that run a ranked list (a restaurant feed or text search) want to compare a new ranker with the current one without waiting for an A/B test. A/B tests are slow to reach power and cost money when the variant shows poor results. Offline evaluation on old logs is biased by selection, presentation and position effects, so its verdict often differs from the online one. Inverse propensity scoring (IPS) corrects this bias but needs a stochastic logging policy that sometimes shows non-top items, and that randomness costs real revenue.

Example from the article: Just Eat Takeaway.com ranks restaurants with deterministic rankers and wanted to predict the click-through rate (CTR) of a new ranker from the old ranker's logs.

Solution in brief

Treat the position, not the ranker, as the source of randomness. A user examines each list position with some probability, and that probability can be estimated from logged clicks. Use the ratio of examination probabilities at the new and the old position as the IPS weight. The result is an offline CTR estimate (OPE) that tracked the A/B-test CTR day by day in two experiments.

Questions this case answers
  • How to estimate offline the click rate a new ranked-list policy would get when the logging policy is deterministic and gives no item selection probabilities
  • How to evaluate a new ranking recommender offline when the production ranker is deterministic and logged data has no randomization?
  • How to get propensities for inverse propensity scoring without paying for a stochastic logging policy?
  • How to check that an offline estimator tracks the real A/B-test result of a ranker?

Tools and methods

  • Inverse propensity scoring (IPS) — reweights logged clicks to estimate a new policy's value
  • Position-bias examination model — gives the probability that a user examines each list position
  • Expectation maximization (EM) — fits the examination probabilities from logged clicks
Continue reading — it's free

Google or a one-time email code. No password.