Encouragement Designs and Instrumental Variables for A/B Testing

Benjamin ElbersAug 2023

Problem

A standard A/B test needs a control group without the feature. That is not always acceptable: the feature may already be live for everyone, a campaign may point users to it, users may share it with each other, or users expect it, like Spotify's annual Wrapped. Comparing users who used the feature with those who did not is biased, because engaged users choose to try new features.

Example from the article: a new feature is enabled for all users, but only a randomly chosen treatment group sees a Home page banner inviting them to try it.

Solution in brief

Randomize an encouragement to use the feature, not access to it. The effect of the encouragement on the outcome, divided by its effect on feature usage, gives the feature's effect for the users the encouragement moved (the compliers). This instrumental variables (IV) estimator needs three assumptions: no defiers, no direct effect of the encouragement, and a working encouragement.

Questions this case answers
  • How to estimate the causal effect of using a feature when the feature cannot be withheld from a control group and usage is self-selected
  • How to recover the effect on users who respond to a nudge when only the nudge, not the treatment, can be randomized
  • How to measure a feature's effect when every user must get access to it and a holdout group is not possible?
  • How to estimate the effect of feature usage when users who choose to use it differ from those who do not?

Tools and methods

  • Encouragement design — random nudge to use the feature, while the feature is open to all
  • Instrumental variables (IV) estimator — turns the effect of the nudge into the effect of using the feature
  • Local average treatment effect (LATE) — effect for compliers only (also called CACE)
  • Sensitivity analysis — shows how a small direct effect of the nudge would move the LATE
Continue reading — it's free

Google or a one-time email code. No password.