Problem
Lyft's decision problems, such as dynamic pricing and recommendations, have no labeled "correct answer". Supervised and optimization models struggled with them: a supervised model only predicts, so thresholds or business logic must be added to turn a prediction into a decision. The environment also drifts (customer preferences, competition, a pandemic). Contextual bandits fit these problems, but few mature libraries and little guidance existed.
Example from the article: a demo news recommender picks one of four article categories from two context features (user and time of day) and gets reward 1 for the preferred category and 0 otherwise.
Solution in brief
Lyft extended its supervised-learning platform instead of building a new one, reusing serving, the model registry and the retraining CI/CD pipeline. The RL-specific changes are small: pass the action space with each request, log the action propensities, and join business events into a reward for incremental updates. Bandit models use Vowpal Wabbit and are developed offline with the Coba off-policy evaluation framework.
Questions this case answers
- How to choose actions per context and learn from partial feedback when no labeled ground truth exists and the optimization model is too rigid
- How to estimate the performance of a new decision policy from data logged by a different policy
- How to serve and incrementally retrain a decision policy that must adapt to a changing environment
- How to run contextual bandit models in production on top of an existing supervised-learning platform?
Tools and methods
- Contextual bandits (Vowpal Wabbit) — choose an action per context, learn from the chosen action's reward
- Lyft RL library — adapts bandit models to the supervised-model interfaces
- Coba — offline benchmark with IPS, DM and DR reward estimators and rejection sampling
- Shapley values — feature importance for choosing context features
- Model CI/CD — scheduled incremental training and promotion
Google or a one-time email code. No password.