Problem
Pinterest teams have many ideas that change the order of content, such as new ranking or blending functions. An online A/B test needs at least seven days, allows only a few variants at once and requires production-quality code. It also pushes teams toward safe ideas. Offline metrics such as Mean Average Precision (MAP) or AUC turned out to be poor predictors of online results, so they do not help to pick what to test.
Example from the article: Home feed ranking and blending models were compared on 26 control versus treatment pairs, both offline and in online experiments.
Solution in brief
Offline Replay Experimentation simulates a test on historical logs. It rebuilds the list the new ranking would have served for each past request, then counts reward only on records that match what production showed, weighted by the inverse of their sampling probability to remove selection bias.
Questions this case answers
- How to predict the online lift of a change in the order of served items from logged data of a different serving policy, when waiting for a randomized test is too slow or costly
- How to correct selection bias when only the items shown by the current policy have observed feedback
- How to predict the result of an online experiment on a ranking or blending change without running it?
- How to choose, from many ranking ideas, the few worth an expensive A/B test?
Tools and methods
- Counterfactual serving simulation — rebuilds what a new ranking or blending function would have served in the past
- Top K Match estimator — sums inverse-probability-weighted reward over records matching on request, position and item
- Top K Unsorted Match estimator — relaxed variant matching on request and item only; higher match rate, lower variance
- Random bucket — small traffic slice with a shuffled top N, used when sampling probability is not logged
Google or a one-time email code. No password.