Lyft’s Reinforcement Learning Platform

Jonas Timmermann12 Mar 2024

Problem

Lyft's decision problems, such as dynamic pricing and recommendations, have no labeled "correct answer". Supervised and optimization models struggled with them: a supervised model only predicts, so thresholds or business logic must be added to turn a prediction into a decision. The environment also drifts (customer preferences, competition, a pandemic). Contextual bandits fit these problems, but few mature libraries and little guidance existed.

Example from the article: a demo news recommender picks one of four article categories from two context features (user and time of day) and gets reward 1 for the preferred category and 0 otherwise.

Solution in brief

Lyft extended its supervised-learning platform instead of building a new one, reusing serving, the model registry and the retraining CI/CD pipeline. The RL-specific changes are small: pass the action space with each request, log the action propensities, and join business events into a reward for incremental updates. Bandit models use Vowpal Wabbit and are developed offline with the Coba off-policy evaluation framework.

Questions this case answers
  • How to choose actions per context and learn from partial feedback when no labeled ground truth exists and the optimization model is too rigid
  • How to estimate the performance of a new decision policy from data logged by a different policy
  • How to serve and incrementally retrain a decision policy that must adapt to a changing environment
  • How to run contextual bandit models in production on top of an existing supervised-learning platform?

Tools and methods

  • Contextual bandits (Vowpal Wabbit) — choose an action per context, learn from the chosen action's reward
  • Lyft RL library — adapts bandit models to the supervised-model interfaces
  • Coba — offline benchmark with IPS, DM and DR reward estimators and rejection sampling
  • Shapley values — feature importance for choosing context features
  • Model CI/CD — scheduled incremental training and promotion
Continue reading — it's free

Google or a one-time email code. No password.