Problem
Airbnb needed to measure the benefit of a new tool that automates part of the workflow of its operations agents. Random assignment was not possible for organizational reasons and fairness concerns. Operational sites differ by country, headcount and skill level, so assigning whole sites to treatment would compare apples to oranges. There are millions of customers but only a limited number of agents, so even a randomized test would lack statistical power.
Example from the article: estimating how productive operations agents would have been without the workflow automation tool, using only agents who did not adopt it.
Solution in brief
Train an ML model on the untreated holdout group to predict the outcome, apply it to the treated group, and take actual minus predicted as the impact. The raw estimate is biased, so Airbnb measures the bias on repeated A/A tests, subtracts it, and reads confidence intervals and p-values off the same A/A distribution.
Questions this case answers
- How to estimate the causal effect of a treatment on a small, non-randomizable population when matching or synthetic control are too crude
- How to build a debiased ML counterfactual and an empirical confidence interval from placebo (A/A) tests
- How to measure the effect of a tool or change when treated and untreated units cannot be randomized?
- How to get a valid confidence interval and p-value for an effect estimated with a machine learning model?
Tools and methods
- ML counterfactual prediction — any model trained on untreated units to predict what treated units would have produced without treatment.
- A/A tests on the holdout — estimate the model's systematic bias and build empirical confidence intervals and p-values.
- Bootstrap — 100 bootstrapped A/A samples give the empirical distribution of null estimates.
Google or a one-time email code. No password.