Optimizing at the Edge: Using Regression Discontinuity Designs to Power Decision-Making

Tilman Drerup, Levi Boxell, Robert Fletcher15 Apr 2024

Problem

Product rules often switch the user experience at a numeric cutoff: arrival time, order amount, a score. The team then has to decide whether to move the cutoff, which means weighing extra orders against extra cost. Comparing users on the two sides of the rule does not work, because who lands where is not random.

Example from the article: Instacart places a card authorization hold equal to the cart total plus a buffer. A higher hold lowers the risk of unpaid amounts but confuses customers and can cause card declines.

Solution in brief

Find the cutoff where a rule makes the treatment jump, and compare units just below and just above it. Units this close are similar on average, so the gap in the outcome is the causal effect for them. Plug the effects on each metric into one inequality to decide whether the cutoff should move. Then confirm the new policy in an A/B test.

Questions this case answers
  • How to estimate the causal effect of a rule that switches treatment at a numeric cutoff when the assignment variable is not random but units near the cutoff are comparable
  • How to turn local effect estimates on several metrics into a go/no-go decision on moving a threshold
  • How to estimate the effect of a threshold rule on business metrics without running an A/B test?
  • How to decide whether to move a threshold when a change brings more orders but also more cost?

Tools and methods

  • Regression discontinuity design (RDD) — estimates the local effect of a threshold rule from observational data.
  • Local average treatment effect (LATE) — the effect for units near the cutoff only.
  • Trade-off inequality with a guardrail — turns effects on orders and cost into a go/no-go decision.
  • A/B experiment — validates the revised policy and gives the average effect across all users.
Continue reading — it's free

Google or a one-time email code. No password.