Method
Inverse propensity scoring
IPS estimator reweighting logged feedback by action propensity.
8 cases from 8 companies apply Inverse propensity scoring. 5 were checked in an online A/B test.
Typical tasks: Evaluate models, Rank search results, Recommend to new users and new items, Measure impact without an A/B test and 1 more. Companies: Netflix, Avito, Sber, Lyft, Instacart, Thumbtack and 2 more. Often combined with: Doubly robust estimation, Off-policy evaluation, Gradient boosting, Multi-armed bandit, SHAP and 17 more.
| MAPS: Netflix’s Multimodal Asset Personalization at Scale | media-streaming | 2026-08 | |
| Models in Monetization | classifieds | 2026-05 | |
| The New Policy Hasn't Run Yet: Comparing It to the Old One Before an A/B Test | banking | 2026-05 | |
| Lyft’s Reinforcement Learning Platform | mobility | 2024-03 | |
| Using Contextual Bandit models in large action spaces at Instacart | e-commerce | 2023-06 | |
| Evolution of Search Ranking at Thumbtack | marketplace | 2022-12 | |
| Inverse Propensity Score based offline estimator for deterministic ranking lists using position bias | food-delivery | 2022-08 | |
| Experiment without the wait: Speeding up the iteration cycle with Offline Replay Experimentation | social | 2022-01 |