Real-World Forecasting with Deep Learning: How We Do It at Wix

Ariel Berger2024

Problem

Wix forecasts collections income far ahead, and stakeholders plan on it. Patterns drift with seasons, markets and one-off events, so the model needs frequent retraining. Transformer-based models train slowly. Their long-horizon forecasts change a lot between retrains even when accuracy looks stable. A simulation in the post shows that small weekly changes can add up to a $15M week-on-week swing in the forecast, which makes planning hard.

Example from the article: collections that depend on planned sale days and new-user sign-up trends, with weekly retraining.

Solution in brief

Wix uses TiDE, an MLP-based forecaster that trains much faster than transformers and accepts external covariates. Retraining is weekly and the published forecast is the average of the last 4 models. The last part of the training set doubles as validation, and hyperparameters are tuned with Optuna over rolling cross-validation. The team reports a 40% improvement in the collections income forecast.

Questions this case answers
  • How to make a periodically retrained long-horizon forecast stable enough that stakeholders keep trusting it while it still reacts to new trends
  • How to tune and validate a neural time-series model when rolling-origin cross-validation is too slow and the latest data is reserved for validation
  • How to keep a deep learning forecast of a business metric current without retraining costs getting out of hand?
  • How to stop long-horizon forecasts from jumping every time the model is retrained?

Tools and methods

  • TiDE (Time-series Dense Encoder) — encoder-decoder forecaster built from MLPs, supports exogenous covariates, fast to retrain
  • Ensemble of the last 4 retrained models — averaging smooths forecast revisions and may add accuracy
  • Train-tail validation — the final part of the training set is also the validation set
  • Rolling cross-validation (forward chaining) — monthly backtests across a year to score a hyperparameter set
  • Optuna — Bayesian hyperparameter search that cuts the number of training runs
Continue reading — it's free

Google or a one-time email code. No password.