Measuring Product Impact Without A/B Testing: How Discord Used the Synthetic Control Method for Voice Messages

Alec Brevé, Angela Ambroz27 Nov 2024

Problem

Discord wanted to measure the effect of Voice Messages in its mobile app. The feature works only when one user sends a message and another receives it. A user-level A/B test leaks: a treated sender reaches an untreated friend, and either the friend is exposed to the treatment or the sender's experience changes. The platform also did not support cluster randomization. A server-level test confused users. A geo test against one lookalike country mixes the feature effect with every other difference between the two countries.

Example from the article: in 2023 Discord added Voice Messages to its mobile app and released it in Brazil as the treated country, without a randomized control group.

Solution in brief

Release the feature in one country and compare it to a "synthetic" version of that country: a weighted mix of countries without the feature, fitted so that it tracks the treated country before launch. The gap after launch is the estimated effect. A country is a rough network boundary, because most networks are country- or language-specific, so network effects are mitigated.

Questions this case answers
  • How to estimate the causal effect of a change on a metric when treated and untreated units interact and randomization by unit or by cluster is not available
  • How to build a counterfactual for one treated unit from a weighted mix of untreated units and check its quality
  • How to measure the effect of a feature whose value depends on other users receiving it, when user-level A/B tests are contaminated?
  • How to evaluate a feature rolled out in one country without comparing it to a single lookalike country?

Tools and methods

  • Synthetic control — builds the counterfactual for the treated geo from weighted untreated geos.
  • Mean squared prediction error (MSPE) — checks how well the synthetic control fits before and after launch.
  • Synth (R) / SyntheticControlMethods (Python) — libraries that fit the weights.
Continue reading — it's free

Google or a one-time email code. No password.