Problem
Airbnb's forecasting team produces demand, booking and cancellation forecasts that other teams plan around. When one drifts, the default reaction is a retrain on a fixed monthly or quarterly cadence. The cadence always picks a refit, so the choice between refitting, changing the model and doing nothing is rarely made on purpose. A refit on unusual data can degrade a good model, and it cannot remove a stale assumption, so bias persists for quarters and spreads downstream.
Example from the article: a forecast missed in the same direction for a couple of quarters through several routine refreshes. A pandemic-era assumption about cancellation timing stayed on after behavior normalized.
Solution in brief
Treat "retrain" as three different actions: refit (same model, new data), respecify (change the structure, priors or likelihood) or hold (leave the model alone). Choose by what changed: parameters drifted, the process changed in a way the model cannot represent, or the miss is within the normal range. The signature of the miss (direction, and what fresh data does to it) tells which case it is.
Questions this case answers
- How to decide whether a degrading production forecast needs new data, a new model structure, or no change when retraining runs on a fixed schedule
- How to detect that forecast errors come from a changed data-generating process rather than from parameter drift
- How to decide whether a drifting forecast needs a refit, a respecification or no change?
- How to tell a structural break from ordinary parameter drift in a production forecast?
Tools and methods
- Refit / respecify / hold decision rule — three questions asked in order on every miss.
- Residual direction check — a one-sided miss that survives refits points to structure, not parameters.
- Bayesian hierarchical forecast with a pooling prior — the model class in the post's examples; priors can borrow strength along chosen links.
Google or a one-time email code. No password.