Are Vaccines Effective? Looking at Publicly Available Data

Ivan Komarov, Gleb Sosnin, Egor Kobylkin4 Jun 2022

Problem

An analyst has public country-level data and a question about an intervention nobody randomized, here COVID-19 vaccination. A plain correlation looks convincing: more vaccinations per 100 people go with lower excess mortality. But filtering to rows with both variables shifts the sample (median age, hospital beds per capita), and averaging favors countries with more observations. The team wanted a causal estimate instead.

Example from the article: 22 similar countries from Our World in Data (OWID) that vaccinated at different speeds, with mortality compared 90 days before and after each reached 10%, 25% and 50% vaccinated.

Solution in brief

Build pairs of countries that look alike in wealth and epidemic course but differ in vaccination speed. For each country take the change in mortality around a vaccination milestone, then subtract one country's change from the other's (difference-in-differences, DiD). The result was inconclusive, but the pair-selection recipe and its traps transfer to any staggered rollout.

Questions this case answers
  • How to estimate the effect of an intervention rolled out at different speeds across units when only public aggregate data exists and nothing was randomized
  • How to choose comparable control units for a pairwise difference-in-differences when pre-period outcome trends cannot be used for matching
  • How to estimate the effect of an intervention from public aggregate data when there is no randomization?
  • How to pick comparable units (countries) for a difference-in-differences comparison?

Tools and methods

  • Difference-in-differences (DiD) — removes the shared time effect and fixed unit differences
  • KMeans on country covariates — three prosperity clusters (poor, moderate, rich)
  • Pair selection by similarity thresholds — similar epidemic course and cluster, different vaccination speed
  • Regularized regression — "generalizing" model of deaths on country, period and vaccination features
  • Our World in Data (OWID) — public aggregator of tests, beds, GDP, vaccination and mortality
Continue reading — it's free

Google or a one-time email code. No password.