Experiment Faster and with Less Effort

Yicong (Nicole) Lin, Yixin Tang13 Feb 2024

Problem

The consumer retention marketing team at DoorDash tests CRM campaign policies on user segments. Each extra policy needs its own randomized segment, set up by hand across several platforms, so effort grows linearly with the number of policies. Every policy also needs a minimum sample size under a limited marketing budget. Retention-type metrics are slow to measure, so only a few policies get tested per quarter, and there is too little data for personalized (heterogeneous effect) models.

Example from the article: a promo policy built from four components (promo spread, discount, triggering timing, messaging) gives 24 combinations, too many to launch at once.

Solution in brief

Split the policy into factors with levels and assume no interaction effects between them. Then a fractional factorial design needs only 8 of the 24 variants, plus one extra out-of-sample variant that checks the assumption. Effects of all 24 policies are derived from the 8, and an HTE model can be added on top.

Questions this case answers
  • How to compare many combinations of multi-level policy components when each implemented variant costs setup effort and sample size
  • How to validate with data the simplifying assumption that makes a reduced experimental design valid
  • How to test many combinations of promo policy components in one experiment without building every variant?
  • How to cut the setup cost and sample size of a CRM campaign test with many arms?

Tools and methods

  • Fractional factorial design — run a subset of factor-level combinations and estimate factor effects from it
  • Out-of-sample validation variant — a ninth arm outside the design that tests the no-interaction assumption
  • Heterogeneous treatment effect (HTE) model — finds user-specific best policies when a joint test shows personalization helps
Continue reading — it's free

Google or a one-time email code. No password.