Problem
A streaming service wants recommendations that raise long-term satisfaction, not only clicks or plays in the session. Retention looks like the natural target, but it is noisy, sensitive only for members close to cancelling, hard to attribute to specific recommendations, and arrives once per account per month. Optimizing click-through alone promotes clickbait. A further problem is that useful feedback, such as finishing a season or a thumbs-up, comes weeks later or never, while the policy must be updated quickly.
Example from the article: a member plays a recommended show, completes it and gives a thumbs-up. A simple proxy reward is then .
Solution in brief
Treat recommendation as a contextual bandit and train it on a proxy reward that is aligned with long-term satisfaction yet reacts to each single recommendation. Refine the reward in a loop (hypothesis, new reward, new policy, A/B test). Fill in delayed or missing feedback with a separate model that predicts it from the feedback observed so far.
Questions this case answers
- How to optimize a recommendation policy for a long-term outcome that is measured once a month and depends on many outside factors
- How to compute a training reward from feedback that is delayed or missing at the time the policy must be updated
- How to define a reward for a recommender that tracks long-term satisfaction when retention is too slow and noisy to optimize
- How to train a bandit policy when part of the user feedback arrives weeks later or never
Tools and methods
- Contextual bandit — policy that picks recommendations for a member context and learns from reward
- Proxy reward — function of play, completion, thumbs and similar signals that stands in for retention
- Reward engineering — iterative redesign of the reward, like feature engineering
- Delayed feedback prediction model — predicts final feedback from feedback observed so far, used offline for reward computation
Google or a one-time email code. No password.