Problem
Data scientists increasingly hand analysis to LLM agents. For observational causal questions this is risky: the agent runs a regression and returns a polished number, with no sign of whether it handled confounding. Real data has no ground truth, so the human cannot tell a right answer from a wrong one.
At Netflix the question was how a new entertainment type affects member retention. The first users of any new offering differ from the average member, so a naive estimate is biased.
Example from the article: A plain regression of two-month retention on days engaging with "Type X" gave an estimate four times larger than the scaffolded workflow, because early adopters were heavier, more passionate users.
Solution in brief
Run the agent inside a fixed scaffold: a templated notebook with doubly robust estimation, four mandatory design diagnostics, and an actor-critic loop. The actor runs the analysis and fixes failed diagnostics by a playbook. The critic grades credibility and states how the estimand changed. All outputs are inspectable artifacts for a human.
Questions this case answers
- How to estimate a treatment effect from observational data when early adopters differ systematically from everyone else and no experiment can be run
- How to make an automated analysis agent follow a rigorous causal-inference checklist so a human can audit each step without ground truth
- How to estimate the causal effect of a new offering on retention without an A/B test?
- How to make an LLM agent run observational causal analysis without silently returning a biased number?
Tools and methods
- Doubly robust (AIPW) learning in EconML — effect estimation in the templated notebook.
- Four design diagnostics — covariate balance, overlap, placebo outcome, sensitivity to hidden confounders.
- Propensity trimming — restrict to the overlapping population when overlap fails.
- Actor-critic agent loop — actor executes, critic reviews, human (principal) supplies the plan and audits.
- Claude Sonnet 4.6 — the model behind baseline and agent.
Google or a one-time email code. No password.