Problem
Cian is a two-sided real estate classified. Its monetization team often tests features that change ranking, such as an auction for listing placement. In a user-level A/B test the control group also feels the effect, because its listings get pushed down in search, so the SUTVA assumption breaks. A regional experiment isolates sellers and their search results. It does not help when a feature goes to pre-selected users with no link to a region, and synthetic diff-in-diff cannot be applied there.
Example from the article: a change to the conditions for sellers who hold listing packages, for instance fewer free posting slots, where the affected users are not defined by region.
Solution in brief
Estimate the effect with matching on pre-period confounders. Choose confounders by causal reasoning, with no mediators or colliders. Choose the matching type by balance metrics measured on history. Pick the test group from the options the business gives, or by automatic optimization with Optuna.
Questions this case answers
- How to estimate the effect of a change that cannot be randomized at unit level because treated and control units interfere, using pre-period covariates to build comparable groups
- How to choose adjustment variables for matching so that mediators and colliders are excluded and balance is measured on history
- How to estimate the effect of a feature that cannot be randomized at user level because of network effects
- How to estimate the effect of a feature rolled out to pre-selected users with no link to a region
Tools and methods
- Matching (nearest neighbor, optimal pair, optimal full, fast full, CEM, profile matching) — builds a control group balanced on confounders
- Balance metrics (SMD, Max/Mean ECDF, variance ratio, QQ-plot) — compare matching types and parameters
- Brute-force bias estimation on pre-period data — double-checks the confounder list
- Optuna — optimizes the choice of test regions or users
Google or a one-time email code. No password.
