Problem
A user opens a homepage with a slate of recommendations and has limited time to choose. Each item costs time to evaluate: recognizing the box art, watching a trailer, reading the synopsis. A slate ranked by relevance alone can put expensive items first, so the budget runs out before the user reaches items they would have played. The user's budget, like their taste, is not observed.
Example from the article: a user arrives at the Netflix homepage looking for something to watch. A popular show such as Stranger Things needs almost no evaluation time, while a less known title needs much more.
Solution in brief
Model slate building as a Markov decision process (MDP): the recommender adds one item per slot, and the state holds the user's remaining time budget. Train a value function with temporal difference learning and use it as the item scorer in the usual bandit-style slate pipeline. With discount factor this is a contextual bandit; with it is reinforcement learning (RL).
Questions this case answers
- How to construct an ordered list of items that maximizes the chance of a positive response when the consumer can only examine a limited number of them before leaving
- How to trade off item relevance against the per-item cost of evaluation when the consumer's budget is hidden
- How to build a recommendation slate when the user has only a limited amount of time to look at it?
- How to trade off item relevance against the time it takes the user to evaluate the item?
Tools and methods
- Markov decision process — recommender is the agent, user is the environment
- TD(0) / SARSA — on-policy value function used as the item scorer
- Q-learning — off-policy variant for logs from another policy
- Contextual bandit — the baseline
- 0/1 knapsack — the view of the problem as choosing items under a cost budget
- Simulation — generative user model for comparing agents
Google or a one-time email code. No password.