Geo-based AB Testing and Difference-in-Difference Analysis in Instacart Catalog

Xiaoding Krause, Kriston Costa2 Jun 2022

Problem

Instacart's Catalog team owns the warehouse layer that publishes product attributes to the shopping app, fulfillment and ML teams. To test a new attribute on user outcomes, every consuming team must add an index for it. The alternative is to duplicate the whole catalog of billions of attributes for each experiment. Both cost too much to scale, so major catalog projects lacked causal evidence.

Example from the article: testing an ML-processed product name, judged by user orders and purchases.

Solution in brief

Randomize delivery zones instead of users. The new attribute is published only for treatment zones' inventory areas, before product data is localized to stores, so no downstream team changes anything. Similar zones are matched on pre-period features, and the effect is read with a two-way fixed effects difference-in-differences model with clustered errors. Daily ETLs and a web UI automate setup and analysis.

Questions this case answers
  • How to estimate the causal effect of an upstream data change when user-level randomization would require every consuming system to be rebuilt
  • How to choose a few comparable clusters for randomization when units are heterogeneous and users leak between clusters
  • How to A/B test a change in an upstream data pipeline when every downstream team would have to build new indexes to show two versions to users?
  • How to pick treatment and control geographic zones that are similar enough to trust a difference-in-differences readout?

Tools and methods

  • Geo experiment — delivery zones are the randomization unit.
  • Nearest-neighbor matching with a greedy algorithm — groups similar zones, one per variant.
  • PCA — compresses zone features before distances are computed.
  • Difference-in-differences with two-way fixed effects — estimates the treatment effect.
  • A/A test and test of equivalence — validates the parallel-trend assumption.
  • Power analysis — sizes the test with clustered errors.
  • Holdout — zones kept on the old catalog to measure cumulative impact.
  • linearmodels, Apache Airflow — clustered two-way fixed effects and daily pipelines.
Continue reading — it's free

Google or a one-time email code. No password.