Problem
Walmart forecasts thousands of time series for planning and resource allocation. Fitting and validating a SARIMAX, Holt-Winters or Prophet model for every series takes too long, and running that many models in production is expensive to maintain. One regressor over all concatenated series with a one-hot series identifier tends to miss trend, seasonality and autocorrelation without hand-made time-series features. Multivariate methods such as Vector Autoregression (VAR) rest on strong parametric assumptions and scale poorly.
Example from the article: a cluster of 244 monthly series, 56 months of history each, forecast for the next 12 months; the same approach was applied to more than 50 thousand series in one Walmart forecasting project.
Solution in brief
Cluster the series by temporal pattern, then fit one ordinary univariate model to each cluster's median series. Each series' own offset from the median is forecast by a very simple model, and the two forecasts are added. The article reports a 20x cut in fitting time and a 500x cut in the number of models, with accuracy close to per-series models.
Questions this case answers
- How to produce forecasts for thousands of related time series when fitting and maintaining one model per series is too slow and costly, and a generic regressor loses trend and seasonality
- How to forecast thousands of similar time series without fitting a separate model for each one?
- How to keep trend and seasonality in a forecast when one model serves a whole cluster of series?
- How to cut model training time and the number of models to maintain in production for large-scale forecasting?
Tools and methods
- Bisecting k-means — groups series with similar temporal patterns, Euclidean distance on the series values
- Prophet — univariate model fitted to the cluster's median series (any univariate model can take its place)
- Linear regression — per-series model of the adjustment series
Google or a one-time email code. No password.