Trusting the Untestable: Validation and Diagnostics for the Doubly Robust Models

Shima Nassiri, Ross Chu17 Feb 2026

Problem

Lyft cannot randomize some decisions: partnerships between two companies, long-term effects of high prices, or policies already rolled out. It estimates these effects from observational data with augmented inverse propensity weighting (AIPW). Without randomization, nothing guarantees an unbiased estimate. Selection bias, extreme propensity weights and hidden confounders can each bias it.

Example from the article: drivers who get a weekly ride challenge (a bonus for completing a set number of rides in a week) are chosen by a targeting algorithm. AIPW on this observational data showed an 11.1% lift in driver hours, while a randomized benchmark showed 13.3%.

Solution in brief

Make validation part of the platform: a mandatory pre-exposure confounder set, overlap and balance diagnostics with every result, and a one-time benchmark against a randomized experiment. The benchmark traced the gap to propensity trimming, so two more diagnostics were added: a hidden-confounding sensitivity metric and a trimmed-versus-untrimmed covariate comparison.

Questions this case answers
  • How to validate an observational causal estimate when randomization is impossible and unobserved confounding cannot be tested directly
  • How to benchmark an observational estimator against a randomized experiment when the experiment only randomizes eligibility, not the treatment itself
  • How to trust an AIPW (doubly robust) effect estimate when the treatment cannot be randomized?
  • How to check overlap and covariate balance before reporting an observational effect?

Tools and methods

  • AIPW — doubly robust ATE or ATT from an outcome model and a propensity model
  • Overlap histogram and trimming — check common support, drop extreme scores
  • Covariate balance plot — compare groups before and after weighting
  • Instrumental variables (LATE) — experimental ground truth when assignment does not fully determine treatment
  • Marginal Sensitivity Model (Zhao et al., 2018) — robustness to hidden confounders
Continue reading — it's free

Google or a one-time email code. No password.