Problem
A marketplace team analyzes experiments where treatment is assigned to geographic regions, or to region and time slices. Controlling for region, date or shopper effects improves precision, but each group level needs a dummy variable. With tens of thousands of levels, ordinary least squares (OLS) runs out of time and memory. Without those controls, static-geo tests are underpowered: their effective sample size is an order of magnitude smaller than in a switchback.
Example from the article: Instacart's Marketplace team ran a static-geo test that removed one delivery-window option and measured order fulfillment time ("active time"). A plain bivariate regression gave a standard error of 285% of the effect.
Solution in brief
Demean the outcome and regressors by each fixed effect instead of adding dummies. For several fixed effects, repeat the demeaning in turn until the residual group means fall below a tolerance (the method of alternating projections). PyFixest and fixest do this by default and keep run time and memory low.
Questions this case answers
- How to estimate a treatment effect with fixed effects for many groups when dummy-variable regression no longer fits in time or memory
- How to raise the power of an experiment whose unit of assignment is a small number of large clusters that never switch treatment
- How to analyze a marketplace experiment with geo and time fixed effects when the number of groups is in the tens of thousands?
- How to get enough statistical power in a static-geo test whose effective sample size is an order of magnitude smaller than a switchback?
Tools and methods
- PyFixest / fixest — high-dimensional fixed-effects regression via demeaning
- Frisch-Waugh-Lovell theorem — justifies replacing dummies with demeaning
- Method of alternating projections — handles several fixed effects at once
- Geo:time switchback and static-geo designs — the two experiment designs the team uses
- Difference-in-differences with two-way fixed effects — the estimator for static-geo tests
Google or a one-time email code. No password.