Problem
Instacart gives some users free trials or discounted annual memberships (Instacart+) and must decide who gets an incentive and how large it is. The goal is to maximize long-term value (LTV), but an A/B test runs for weeks while the payoff arrives over years. A discount can lose money first and pay back only through retention and renewals. One short-run metric such as month-1 revenue cannot rank incentives that work through different behaviors.
Example from the article: one initiative raises order size (revenue up, orders flat) and another raises order frequency in exchange for temporary price cuts (revenue down, orders up). Month-1 revenue alone cannot project the LTV of both.
Solution in brief
Train a surrogate index on past experiments: a model that maps several short-run outcomes and pre-treatment features to the long-run outcome. Predict each user's monthly profit trajectory: a flexible model for early months, a one-parameter growth trend afterwards. Validate by comparing predicted effects with actual long-run effects in old experiments, overall and by segment.
Questions this case answers
- How to estimate the long-run effect of an intervention from short-run experiment data when the long-run outcome is not yet observed
- How to validate a long-horizon effect estimator when only medium-horizon ground truth exists
- How to estimate the long-run value of a membership discount when A/B tests last only weeks?
- How to compare incentives that change different short-run behaviors, such as order size and order frequency, on one scale?
Tools and methods
- Surrogate index — maps month-1 outcomes and pre-treatment features to the long-run outcome
- Experiment library — central store of past incentive experiments, used as training data
- XGBoost — predicts monthly profit levels and the per-user growth parameter
- Growth model — extrapolates profit beyond the observed months
- Backtesting — compares predicted and actual effects on old experiments
Google or a one-time email code. No password.