Problem
Nextdoor ships iOS and Android releases weekly through phased rollouts: 1% of users on day 1, 2% on day 2, and so on. A silent regression in a business metric such as app sessions is hard to see, because dashboards look at aggregates across all users of a platform and a 1% rollout dilutes any effect. Early adopters are also more active than the average user, so their metrics cannot be compared with the rest. The drop usually shows up only at full rollout, days later.
Example from the article: a hypothetical iOS version v1.234.5 fails to count an app session 5% of the time. At 1% rollout this is about 0.05 x 0.01 = 0.05% of all iOS sessions, which is lost in noise.
Solution in brief
Treat each release as a quasi-experiment. Users who adopted the new version on a given day are the treated group, users still on older versions are the control, and a difference-in-differences regression estimates the release effect on each metric. A pre-trend check confirms both groups behaved alike before adoption.
Questions this case answers
- How to detect a small treatment-induced regression early when the treated group is a tiny, self-selected share of users and aggregate metrics are too noisy
- How to estimate a treatment effect without randomization when treated units differ systematically from untreated ones
- How to detect a metric regression caused by a new app version when only 1% of users have it
- How to compare users who adopted a release with users who did not, when early adopters are more active
Tools and methods
- Difference-in-differences (DiD) — estimates the release effect against a counterfactual built from non-adopters.
- Linear regression with a Time_Period x Treated interaction — gives the effect size and its significance per metric.
- Pre-trend check (standard deviation bound) — verifies the two groups behaved alike before adoption.
Google or a one-time email code. No password.