How DoorDash is pushing experimentation boundaries with interleaving designs

Tim Knapik, Stas Sajin27 Aug 2024

Problem

DoorDash search and ranking teams found A/B tests too insensitive for ranker changes. Reliable results needed several weeks of testing on all traffic, so new ideas shipped late and weak changes stayed in front of users for the whole test. Most ranker changes only move results at the margin, and most exposures end without a click or purchase, so the signal is small.

Example from the article: two rankers return nearly the same store list and differ only in the last items, so almost every user click says nothing about which ranker is better.

Solution in brief

Show every user a blend of several rankers' lists, so each user is their own control. Drop exposures that carry no information: items the rankers placed alike and users who did not engage. Attribute clicks, checkouts and order value to the ranker that supplied each item, then compare rankers with a paired t-test. DoorDash reports about 384x average sensitivity over A/B on under 5% of traffic.

Questions this case answers
  • How to compare two ranking algorithms with high statistical sensitivity when a standard between-user randomized test needs weeks of full traffic
  • How to remove non-informative exposures from a within-user comparison without breaking cross-experiment comparability
  • How to detect a small ranker improvement without running a multi-week A/B test on all traffic?
  • How to analyze an interleaved experiment so that users who ignored the list do not dilute the signal?

Tools and methods

  • Interleaving — blends several rankers' lists into one list per user
  • Competitive-pair filtering — keeps only items where rankers disagree
  • Non-engagement removal — drops exposures with no user action instead of counting a tie
  • Paired t-test — compares rankers on per-user aggregates of item-level events
  • Parallel A/B test on reserved traffic — checks latency, which interleaving cannot see
Continue reading — it's free

Google or a one-time email code. No password.