Recommender systems need a user model

Benjamin Heymann, Flavian Vasile26 Sep 2023

Problem

Most recommenders are trained as pattern-recognition or bandit systems. They predict what a user will click or buy, while the way a user decides stays implicit. Teams then cannot say how a recommendation changes a decision, why offline metrics fail to match online A/B results, or how much of a purchase the recommender caused. The same gap makes it hard to measure the incremental value of recommendations and performance advertising.

Example from the article: a shopper in an online bookshop is torn between a novel and a comic. Recommendations, search and a banner each change the path to the final purchase in a different way.

Solution in brief

The authors propose to write down a simple user choice model: a browsing policy that collects information and a consumption policy that decides to buy. A recommender then adds value in two ways, a shortcut effect (less friction) and an information effect (new or reminded information). Incrementality is the extra utility from these two effects.

Questions this case answers
  • How to describe the mechanism by which a decision-support system changes a user's choice, so that its incremental effect can be separated from baseline behavior
  • How to make the implicit user-behavior assumptions of a learned ranking system explicit and testable
  • How to explain what a recommender actually changes in a user's purchase decision?
  • How to separate the incremental effect of recommendations or ads from purchases that would happen anyway?

Tools and methods

  • User choice model — two policies (browsing and consumption) optimizing a utility, with a hidden user type.
  • Shortcut and information effects — a taxonomy of what a recommendation changes in a decision.
Continue reading — it's free

Google or a one-time email code. No password.