Problem
A team builds personalization, for example a contextual bandit that picks a different variant for each user. A contextual bandit looks much like an experimentation tool, so it is tempting to buy one product for both jobs. That mixes two different infrastructure needs, and the personalization system still has to be evaluated as a whole, because no single "best variant" exists to test. Spotify describes how it split the work and why it does not use multi-armed bandits for experiments.
Example from the article: a checkout page shows one Buy Now button to everyone. A contextual bandit now picks the button by cart contents, age and location. To learn whether that beats the static button, the team must still run an experiment against it.
Solution in brief
Build personalization in the ML stack, which serves models at low latency. Evaluate it in the experimentation stack, where the whole bandit or recommender is one treatment variant in an ordinary A/B test or rollout. Integrate the two platforms so that starting such an experiment takes few steps.
Questions this case answers
- How to evaluate an adaptive, per-user decision system when it is itself the product feature being tested
- How to split responsibilities between a model-serving platform and an experimentation platform so that neither limits the other
- How to evaluate a contextual bandit or any personalization system with experiments?
- How to decide whether a bandit belongs in the experimentation tool or in the ML stack?
Tools and methods
- A/B test with guardrail metrics — evaluates every personalization system as a treatment variant
- Contextual bandit — personalization model, lives in the ML stack
- Multi-armed bandit — adaptive traffic allocation, not used at Spotify for experimentation
- Confidence — Spotify's experimentation platform, integrated with the ML platform
Google or a one-time email code. No password.