Problem
Discord wanted to measure the effect of Voice Messages in its mobile app. The feature works only when one user sends a message and another receives it. A user-level A/B test leaks: a treated sender reaches an untreated friend, and either the friend is exposed to the treatment or the sender's experience changes. The platform also did not support cluster randomization. A server-level test confused users. A geo test against one lookalike country mixes the feature effect with every other difference between the two countries.
Example from the article: in 2023 Discord added Voice Messages to its mobile app and released it in Brazil as the treated country, without a randomized control group.
Solution in brief
Release the feature in one country and compare it to a "synthetic" version of that country: a weighted mix of countries without the feature, fitted so that it tracks the treated country before launch. The gap after launch is the estimated effect. A country is a rough network boundary, because most networks are country- or language-specific, so network effects are mitigated.
Questions this case answers
- How to estimate the causal effect of a change on a metric when treated and untreated units interact and randomization by unit or by cluster is not available
- How to build a counterfactual for one treated unit from a weighted mix of untreated units and check its quality
- How to measure the effect of a feature whose value depends on other users receiving it, when user-level A/B tests are contaminated?
- How to evaluate a feature rolled out in one country without comparing it to a single lookalike country?
Tools and methods
- Synthetic control — builds the counterfactual for the treated geo from weighted untreated geos.
- Mean squared prediction error (MSPE) — checks how well the synthetic control fits before and after launch.
- Synth (R) / SyntheticControlMethods (Python) — libraries that fit the weights.
Google or a one-time email code. No password.