Problem
Lyft needs a real-time prediction of whether a rider who has entered a destination and seen a price and ETA will request a ride. The decision depends on many contexts at once: place, time, supply and demand, and the rider's history. Slicing the data that finely leaves few examples per context. A gradient boosted model fitted on 10 sessions with 8 conversions can confidently predict 80%, which is likely noise. Neural or LLM models add too much inference time for live use.
Example from the article: a business traveler leaving the Detroit suburbs at 4:00 AM on a Tuesday for an airport 30 minutes away may match only about ten historical sessions.
Solution in brief
Organize sessions into a tree by a hierarchy of context keys and fit one small parametric model per node, top-down. Each child is fitted with an L2 penalty that pulls its parameters toward its parent's, which is a Gaussian Bayesian prior. Sparse leaves fall back to the parent's behavior, and rich ones follow their own data.
Questions this case answers
- How to predict a binary user decision for thousands of fine-grained contexts when most contexts have only a few observations and predictions must be served with very low latency
- How to make a sparse-data model's output respect known directional relationships between a feature and the outcome
- How to predict whether a user will convert in a very narrow context that has only a handful of historical examples?
- How to serve conversion predictions in real time without a heavy neural network or LLM?
Tools and methods
- Hierarchical tree of partition keys — splits sessions from broad to specific contexts.
- Gaussian prior centered on the parent (L2 penalty) — shrinks a child's parameters toward the parent's.
- Logistic regression per node — a simple model that allows sign constraints.
- Monotonic constraint — keeps predictions consistent with domain knowledge.
Google or a one-time email code. No password.