Problem
Lyft runs many ML models and business metrics, and any of them can degrade. A z-score rule is simple and works in SQL, but it gives too many false positives unless thresholds are tuned for every scenario, and per-use-case rules slowed adoption. A flagged anomaly alone does not say why it happened, so the team cannot choose between retraining and fixing features.
Example from the article: a model's mean prediction was flagged as anomalous on several days, and the cause turned out to be spikes and drops in the input feature request_latency.
Solution in brief
Summarize prediction and feature logs into hourly or daily profiles with whylogs, which turns the problem into ordinary time-series forecasting. Forecast each profile metric with StatsForecast and flag points outside the forecast confidence interval. To explain an anomaly, fit a regressor from feature changes to prediction changes and rank features with Shapley values.
Questions this case answers
- How to detect anomalies in many time series with one general method and no per-series threshold tuning
- How to explain an anomaly in a model output by attributing it to drift in the model inputs
- How to detect anomalies in many model and business metrics without tuning thresholds for each one?
- How to find out which input features caused an anomaly in a model's predictions?
Tools and methods
- whylogs — builds data profiles in one pass
- StatsForecast — fast univariate forecasts with confidence intervals
- Fugue — runs the same Python or Pandas logic locally or on Spark
- SHAP — ranks feature drifts by effect on predictions
- Mode Analytics — dashboards for the detected anomalies
- Slack — soft alerts to the owner team of each model
Google or a one-time email code. No password.