Problem
Most recommenders are trained as pattern-recognition or bandit systems. They predict what a user will click or buy, while the way a user decides stays implicit. Teams then cannot say how a recommendation changes a decision, why offline metrics fail to match online A/B results, or how much of a purchase the recommender caused. The same gap makes it hard to measure the incremental value of recommendations and performance advertising.
Example from the article: a shopper in an online bookshop is torn between a novel and a comic. Recommendations, search and a banner each change the path to the final purchase in a different way.
Solution in brief
The authors propose to write down a simple user choice model: a browsing policy that collects information and a consumption policy that decides to buy. A recommender then adds value in two ways, a shortcut effect (less friction) and an information effect (new or reminded information). Incrementality is the extra utility from these two effects.
Questions this case answers
- How to describe the mechanism by which a decision-support system changes a user's choice, so that its incremental effect can be separated from baseline behavior
- How to make the implicit user-behavior assumptions of a learned ranking system explicit and testable
- How to explain what a recommender actually changes in a user's purchase decision?
- How to separate the incremental effect of recommendations or ads from purchases that would happen anyway?
Tools and methods
- User choice model — two policies (browsing and consumption) optimizing a utility, with a hidden user type.
- Shortcut and information effects — a taxonomy of what a recommendation changes in a decision.
Google or a one-time email code. No password.