Correlation Does Not Imply Causation. So What Does?

Grigory Chernov27 May 2023

Problem

Analysts and business partners read metric changes as causes, and the intuition behind this is often wrong. Typical mistakes: "any gap between correlation and causation is noise", "the cause is just missing from our data", "no correlation means no link", "without strong assumptions nothing can be said". Each one leads to wrong conclusions, and models built on them inherit the error.

Example from the article: the "Sputnik" Twitter account (2021) showed that mortality after the Sputnik vaccine was the lowest, but did not account for time. By April 12, different groups had been vaccinated for different lengths of time, and mortality mostly depends on time.

Solution in brief

Treat the observed distribution as one projection of a data generation mechanism. Several different mechanisms can produce the same data, so ask which mechanism you assume before reading a number. Independence tests on observational data, interventions and quasi-experiments help narrow the choice.

Questions this case answers
  • How to decide whether an observed association between two variables reflects a causal effect when the data are observational and a third variable can reverse the picture
  • How to choose a causal identification method for a pooled versus per-segment metric change when the mechanism that generated the data is unknown
  • How to avoid mistaking correlation for causation when interpreting metric changes
  • How to check whether a pooled number or a per-segment number is the right one (Simpson's paradox)

Tools and methods

  • Simpson's paradox — shows that a pooled estimate and a per-segment estimate can disagree, and the causal story decides which to trust.
  • Intervention — fixing a variable to separate mechanisms that look identical in observational data.
  • Conditional independence tests — narrow the candidates to a class of causal structures using observed data only.
  • Instrumental variables, quasi-experiments, A/B tests — ways to isolate an effect when the confounder cannot be measured.
  • DoWhy, CausalImpact — libraries the speaker recommends for applied work.
Continue reading — it's free

Google or a one-time email code. No password.