Problem
Lyft runs ML models for safety classification, fraud detection and real-time pricing, so a degraded model hurts rider and driver experience and revenue. Model quality usually decays slowly rather than failing loudly, which makes it hard to spot. Causes include wrong units or missing features sent by a caller, a changed upstream feature definition, covariate shift, label shift and concept drift. Before the platform existed, a few teams built one-off monitoring, which duplicated work and gave no central view across hundreds of models.
Example from the article: when demand fell during COVID, ETA models were retrained because they over-predicted ride times. Pricing models that took ETA as input then under-predicted sharply.
Solution in brief
Build monitoring as layers, from cheap and automatic to expensive and owner-configured. Phase one: online Model Score Monitoring and Feature Validation. Phase two: scheduled Anomaly Detection and Performance Drift Detection. Give every model the automatic layers, then make monitoring mandatory.
Questions this case answers
- How to detect gradual quality loss in many deployed models when errors are not deterministic and ground truth is delayed
- How to choose and sequence monitoring techniques for a model fleet under limited engineering capacity
- How to catch silent degradation of production ML models before it hits pricing, fraud or safety decisions?
- How to validate model input features online without adding latency to scoring?
Tools and methods
- Model Score Monitoring — time-series alerts on every emitted model score
- Feature Validation — per-request checks of types, ranges, nulls and sets
- Great Expectations — library that defines the validation rules
- Anomaly Detection — daily z-score checks on aggregates of features and scores
- Performance Drift Detection — daily metrics from joining predictions with ground truth
- Apache Spark — runs the offline drift jobs
Google or a one-time email code. No password.