Using Back-Door Adjustment Causal Analysis to Measure Pre-Post Effects

Sharon Cui2 Jun 2022

Problem

Some changes ship without an experiment: a bug fix, or a feature that regulation or technical limits keep out of A/B testing. A plain before/after comparison is biased, because seasonality, competitor moves, campaigns and other launches shift the metric at the same time. It also gives no confidence level. Difference-in-differences needs a control group with a parallel trend, which a bug-affected group lacks.

Example from the article: DoorDash's Global Search team found a critical bug on mobile web, correlated with a drop in key metrics, and fixed it at once. The team still wanted to measure the fix's effect on mobile web conversion rate.

Solution in brief

Treat the period before the change as "control" and the period after as "treatment". Add the same metric from platforms the change did not touch as covariates. They absorb external shocks common to all platforms, so the remaining pre-post difference is read as the effect. Lift, confidence interval and p-value are computed as in a controlled experiment.

Questions this case answers
  • How to estimate the effect of a change with a confidence interval when randomization is impossible and a pre-post comparison is confounded by simultaneous external factors
  • How to choose a causal method when the treated group already has a distorted trend, so the parallel-trends assumption of a control-group comparison does not hold
  • How to measure the effect of a bug fix or urgent feature that had to ship without an A/B test?
  • How to get a confidence interval for a pre-post metric lift instead of a bare directional read?

Tools and methods

  • Back-door adjustment — blocks confounding paths by adjusting for covariates
  • Experimentation platform — computes the pre-post lift, confidence interval and p-value
  • Instrumental variables, regression discontinuity design — fallbacks when confounders cannot be blocked
Continue reading — it's free

Google or a one-time email code. No password.