Problem
A merchant changes how a product category is organized on the site and wants to know what the change did to browse views and sales. A plain before-and-after comparison mixes the effect with seasonality, trends and other outside factors. A/B tests and two-point difference-in-differences (DiD) assume fairly static conditions. DiD usually compares only two time points, so the onset and the decay of the effect stay hidden.
Example from the article: the ice cream shelves on an e-commerce site are rebuilt into popsicles and bars, ice cream sandwiches, tubs and cones. The merchant wants the causal effect of this taxonomy change on browse views and sales.
Solution in brief
Fit a Bayesian structural time series (BSTS) model on the changed category's history before launch, using unchanged categories as controls. The model forecasts what sales would have been without the change. The gap between actual and forecast sales is the incremental effect, reported with a credible interval for every week after launch.
Questions this case answers
- How to estimate the causal effect of a change on one aggregate time series when only a before-and-after history and unaffected comparison series exist
- How to quantify when an intervention effect appears and how uncertain it becomes as time passes
- How to measure the effect of a site change on sales when no A/B test was run?
- How to build a counterfactual sales forecast from comparable categories that were not changed?
Tools and methods
- Bayesian structural time series (BSTS) — models the pre-launch series as trend, seasonality, control regression and noise, then forecasts the counterfactual.
- Synthetic controls — unchanged categories whose series stand in for the changed category.
- Spike-and-slab prior — picks the useful controls automatically.
Google or a one-time email code. No password.