Problem
A recommender team wants to know how much value personalization creates. The obvious metric, the share of viewing from recommended titles, does not answer this. Titles are recommended because the system expects the member to want them, so a watched recommendation mixes the effect of the recommendation with existing taste. Switching personalization off for a large group would give clean data but harms the member experience.
Example from the article: Netflix needs to separate the appeal of a title from the value of recommending it to the right member at the right time.
Solution in brief
Fit a day-by-day choice model with an explicit bonus for recommended titles, using the natural variation in what the recommender shows. Validate its substitution patterns against small randomized experiments that nudge the visibility of title categories. Then use the model to compare the production recommender with simpler baselines and to split the recommendation effect into exposure, selection and targeting.
Questions this case answers
- How to estimate the causal value of a recommendation policy when items are shown precisely to the users most likely to choose them and disabling the policy is unacceptable
- How to validate a counterfactual choice model against small randomized experiments that shift item visibility
- How to measure the incremental value of recommendations separately from the appeal of the recommended titles?
- How to estimate what viewing would look like under a different recommender without switching personalization off for members?
Tools and methods
- Low-rank choice model — daily choice among catalog titles or no viewing, with learned title and member-state representations
- Recommendation bonus — extra utility for a recommended title, capturing awareness and information
- Randomized visibility nudges — small A/B tests used as ground truth
- Diversion ratios — where viewing goes when a set of titles becomes less visible
- Baselines — random, popularity-based and matrix factorization recommendations for counterfactuals
Google or a one-time email code. No password.