Validating Propensity Score Matching: Metrics and Tests

Ksenia Pevzner24 May 2025

Problem

Teams often need the effect of a feature that cannot be switched off for a random group. At Yandex Go the question was what the premium Ultima tariffs bring. Users already know the service, and offline projects and advertising rule out an experiment. Subtracting Ultima revenue from the total overstates the effect, since some riders would move to cheaper tariffs. Before/after and diff-in-diff comparisons confuse the effect with novelty and other historical factors. Propensity score matching (PSM) can build a control group, but the business doubts its estimates, so the matching must be shown to be sound.

Example from the article: the effect of Ultima rides, Yandex Plus benefits or a new Kinopoisk series.

Solution in brief

Train a classifier to predict who received the treatment and use its probability (propensity score) to pick for each treated user the untreated user with the nearest score. Then validate: score distributions, ROC AUC, covariate balance before and after matching, and the number of unique control users. Check the framework itself on an A/A split and a real A/B test.

Questions this case answers
  • How to estimate the effect of a feature that cannot be randomized by building a matched control group from users who were not treated
  • How to detect overfitting or leakage in a propensity model used to build a matched control group
  • How to estimate the effect of a product feature or tariff when an A/B test is impossible
  • How to check that propensity score matching produced a good pseudo-control

Tools and methods

  • Propensity score matching — pseudo-control by nearest propensity score
  • CatBoost — the propensity classifier; feature weights (fstr) reveal leaks
  • ROC AUC, covariate balance — diagnose overfitting and underfitting
  • A/A and A/B validation — checks the framework itself
Continue reading — it's free

Google or a one-time email code. No password.