Problem
Wildberries & Russ shows targeted ads on screens in order pickup points (PVZ). Sellers choose the targets and the pickup points, so users and points are not randomized, and the groups differ in gender, age, purchasing power, floor area, turnover and item count. Users also move between points with and without screens, and a control user may see someone else's ad. A t-test on raw means is biased, so the ad budget would rest on a wrong number.
Example from the article: a raw difference showed +31.47% purchased items and +36.39% GMV. Propensity score matching cut this to +6.03% and +10.87%, yet it failed a pre-period validation.
Solution in brief
Compare each estimator on a pre-period where no effect exists: the false positive rate (FPR) must stay near 5%, and z-values and p-values must be normal and uniform. Propensity score weighting failed this check. The doubly robust estimator with cross-fitting (double machine learning, DML) passed it with an off-the-shelf Random Forest and gave an effect about half as large.
Questions this case answers
- How to estimate the effect of an intervention from observational data when units cannot be randomized and matching on all features is impossible
- How to choose an effect estimator that keeps the false positive rate near the nominal level, checked on a pre-period where no effect exists
- How to estimate the effect of an ad campaign when users cannot be randomized
- How to control confounders in an observational test when matching on all features is impossible
Tools and methods
- Propensity score + inverse propensity weighting (IPW) — baseline, logistic regression for
- Doubly robust estimator with cross-fitting (DML) — Random Forest for outcome and propensity models
- Pre-period bootstrap validation — FPR, Shapiro–Wilk test of the z-value, Kolmogorov–Smirnov test of the p-value
- Covariate balance and overlap diagnostics — |SMD| below 0.1
Google or a one-time email code. No password.
