Problem
A forecasting team at a delivery marketplace produces forecasts for many granular targets. The usual pipeline trains every model and configuration combination for each target, scores each with rolling-window cross-validation, and keeps the best. Combinations multiply fast, so one run takes hours and hundreds of dollars. For the most granular targets the job does not finish in an acceptable time. A single model is also a poor fit: it assumes one seasonality, is strong only at some horizons, and gives unstable values.
Example from the article: delivery-time forecasts for tens of thousands of switchback units, used as covariates for variance reduction in experiments, which grid search could not complete.
Solution in brief
Instead of choosing one model per target, train many cheap base models once and fit a second-layer model on their out-of-sample forecasts. The second layer learns a weight for each base model. A temporal k-fold scheme produces those out-of-sample forecasts without heavy rolling-window validation.
Questions this case answers
- How to produce accurate forecasts for a very large number of series when exhaustive per-series model and configuration search is too slow and costly
- How to combine many cheap forecasting models so the result beats the best single model without repeated rolling-window validation
- How to forecast tens of thousands of granular targets without running a grid search over model configurations for each one?
- How to get forecasts more accurate than the best single model without picking a winner per target?
Tools and methods
- Stacking ensemble (ELITE) — a second-layer model fitted on base-model forecasts, optionally with external features
- Stacking temporal k-fold cross-validation — builds the training set for the ensemble layer while keeping time order
- statsmodels, Prophet, LightGBM — source of base learners, plus custom models through wrappers
- Ray on Kubernetes (KubeRay) — runs targets and base learners in parallel
- Apache Spark — the earlier backend, kept for comparison
Google or a one-time email code. No password.