How to Determine Causal Effects when A/B Tests are Infeasible through Adopter Analysis

Avanti Chande19 Apr 2024

Problem

An e-commerce subscription business launches a member benefit and wants to know if it raised spend, basket size or retention. A randomized A/B test is the cleanest answer, but sometimes there is no control group, no resources, or the business chose not to test. Quasi-experimental methods (difference-in-differences, regression discontinuity, synthetic control) can then be used, yet they pick up confounders such as tenure, existing engagement, pricing or promotions. Members who use a benefit are often more motivated to begin with, so a raw comparison credits the benefit for what motivation alone would have produced.

Example from the article: an item price discount that a member can redeem on the next purchase, with spend, basket size and retention as outcomes.

Solution in brief

Adopter Analysis combines three steps. Define treated users as adopters (those who actually used the benefit), match each adopter to a non-adopter with a similar pre-launch propensity score, then run difference-in-differences on the matched pairs. Matching makes the groups comparable before launch.

Questions this case answers
  • How to estimate the causal effect of an opt-in treatment on user outcomes when randomization is impossible and users who take it up differ from those who do not
  • How to measure the effect of a new subscription benefit on spend and retention when no A/B test was run?
  • How to build a control group when the users who use a feature differ from those who do not?
  • How to avoid overestimating a treatment effect with plain difference-in-differences?

Tools and methods

  • Adopter definition — treatment group is users who used the benefit at least once, control group is non-adopters
  • Propensity score matching — pairs adopters with similar non-adopters on pre-launch covariates
  • Difference-in-differences (DiD) — compares the pre-to-post change in the matched groups
Continue reading — it's free

Google or a one-time email code. No password.