Forecasting at Databricks: One Framework Behind Consumption and Infra Cost

Ginger Holt, Alex Wang18 Jun 2026

Problem

Data scientists at Databricks cut forecast error by about 80% (measured as WAPE, weighted absolute percentage error), but finance kept publishing the old numbers for six months. A metric like "4% MAPE" (mean absolute percentage error) does not tell a CFO whether to anchor a hiring plan to the forecast, and it cannot answer "why did that number move?". Handing a finished model to the business starts an adversarial loop in which stakeholders distrust even a good forecast.

Example from the article: in infrastructure cost forecasting, hundreds of teams own usage and a few platforms own unit cost, which gives hundreds of thousands of series that no central team can review.

Solution in brief

Ship the decomposition of the forecast as the deliverable; the published number is its roll-up. Build in three properties: explainable (a signed bridge from last month's number), defensible (a backtest win over baselines people already use) and diagnosable (error localized by horizon and vintage). Co-own the forecast with finance and sales; at scale, give each owner a planning app and an AI assistant.

Questions this case answers
  • How to make a forecast that stakeholders adopt by explaining its revisions between runs and proving it beats the baselines they already use
  • How to localize where a forecast error comes from across horizons and forecast vintages without re-auditing the whole model
  • How to get finance to adopt a forecast that is already accurate
  • How to explain why a forecast changed between two monthly runs

Tools and methods

  • Forecast vintage bridge — explains a change between two forecast runs as named, signed components.
  • Backtest against baselines — compares the model with naive, last-year and consensus forecasts.
  • Horizon x vintage error heatmap — locates errors.
  • Databricks Apps, Lakeflow Jobs, Unity Catalog, Metric Views, AI_Forecast(), Genie Code — stack for the app, pipelines and agent.
Continue reading — it's free

Google or a one-time email code. No password.