My Starter Project on the Lyft Rider Data Science Team

Jacob Nogas7 Oct 2025

Problem

Product teams need to know which user experiences (a short pickup wait, an early driver arrival) raise long-term retention, to justify work that improves them. An A/B test usually runs 2–6 weeks, too short for long-term effects, and a feature already rolled out to everyone cannot be tested at all. Comparing exposed and unexposed riders gives a biased answer, because attributes such as region drive both exposure and ride volume.

Example from the article: in a hypothetical low-ETA dataset, a plain comparison shows low ETA lowering rides in the next 28 days, because City B has higher baseline rides and is less often exposed.

Solution in brief

Lyft's Rider Experience Score (RES) tool estimates the effect of an experience on rides in the next 28 days from observational data with the Augmented Inverse Propensity Weighting (AIPW) estimator, using three gradient-boosting models with cross-fitting. The hard part is choosing confounders: keep the key ones and drop any that carry information concurrent with or later than the experience.

Questions this case answers
  • How to estimate the long-term effect of an experience on user retention from observational data when a short randomized test cannot measure it and exposure is confounded by user and context attributes
  • How to select adjustment covariates for a propensity or outcome model so that none carries information concurrent with or later than the treatment
  • How to estimate how a rider experience such as a low pickup ETA changes rides over the next 28 days without running an A/B test?
  • How to avoid a wrong sign in an observational effect estimate when region drives both the experience and the outcome?

Tools and methods

  • Augmented inverse propensity weighting (AIPW) — doubly robust estimate of the average treatment effect (ATE) of an experience on 28-day rides
  • Cross-fitting — data split so each prediction comes from models not trained on that observation
  • XGBoost / LightGBM — models for the two outcome functions and the propensity
Continue reading — it's free

Google or a one-time email code. No password.