How to Accurately Test Significance with Difference in Difference Models

Bradley Doll, Jeffrey Li, Tolga OztanSep 2023

Problem

Spotify often needs the causal effect of a change that cannot be A/B tested, such as an off-platform marketing campaign or the annual Wrapped release. Difference-in-differences (DID) compares the pre/post change of a treated time series with that of a control series. Ordinary least squares (OLS) software reports a p-value for it, but time series are autocorrelated, so that p-value is wrong. In the authors' simulations the false positive rate was over 30% at a nominal 5%.

Example from the article: Spotify promotes one podcast show with marketing spend and uses a second show without a campaign as the control, comparing listening before and after.

Solution in brief

Keep the DID regression, but replace the default significance test with one that respects autocorrelation. Three options are compared by simulation: averaging, clustered standard errors and permutation testing. Clustered errors fit many units, permutation fits few, and averaging is safe but costs much power.

Questions this case answers
  • How to test the significance of a pre/post effect against a control when repeated observations of the same unit are correlated and randomization is impossible
  • How to pick a significance test for a difference-in-differences estimate when the number of comparison units is small
  • How to get a trustworthy p-value from a difference-in-differences model when the data are autocorrelated time series?
  • How to choose between averaging, clustered standard errors and permutation testing for a quasi-experiment?

Tools and methods

  • Difference-in-differences — estimates the effect as the treated pre/post change minus the control pre/post change.
  • Averaging — collapses each unit to one pre and one post point, so ordinary tests become valid.
  • Clustered standard errors — one cluster per time series unit; within-cluster correlation enters the standard error.
  • Permutation test — shuffles treatment and control labels across units to build a null distribution of DID estimates.
Continue reading — it's free

Google or a one-time email code. No password.