Problem
Teams run A/B tests and marketing analyses with p-values, point estimates and bootstrap. None of these answers directly to the business question: how likely is it that the variant pays off, and what is the worst case? They also break down on small groups, unbalanced splits, zero-heavy revenue and nested structure such as stores or regions.
Example from the article: a retailer with stores in different places. Small stores have too little data to be modelled alone, but pooling them with large ones hides real differences between stores.
Solution in brief
Build a white-box Bayesian model by hand: put the known structure (hierarchy, geography, time, causal chain) and rough priors into the model, then read the posterior. The posterior supports statements like "in 95% of cases profit stays above a threshold" and ROI simulation. The talk maps when this pays off; it is not a recipe for one method.
Questions this case answers
- How to estimate effects for small nested groups by building a custom probabilistic model that encodes known hierarchy, geography and time structure
- How to express an experiment result as a risk statement such as the probability that profit stays above a threshold, with a simulated return on investment
- How to decide whether a Bayesian model is worth the effort for a business problem
- How to estimate effects for small groups or new locations with hierarchical models
Tools and methods
- Hierarchical models — share information between groups, so small groups borrow strength from the population.
- Priors — cut absurd parameter values and matter most when data is scarce.
- Posterior ROI simulation — value at risk and best case.
- Simulation-based calibration — pick sample size by simulating data from the prior and refitting.
- Media Mix Model — marketing channel evaluation, with time-varying CAC.
- PyMC — the library, plus PyMC Experimental.
Google or a one-time email code. No password.