Sharpening the Blur: Removing dilution to maximize experiment power

Dave Press, Stas Sajin14 May 2024

Problem

An experiment is diluted when it counts users who could not have been affected by the treatment. They add noise and no signal, so the measured effect shrinks and the test needs far more traffic or time. Assigning every visitor at app entry is easy to implement, which is why dilution is common. At DoorDash it hid the effect in a fraud-model test and a supply-demand test.

Example from the article: a checkout button is changed from blue to green on the web only, but app visitors are also assigned to control and treatment. A 1% lift on web checkout then appears as 0.5% across web and app together, and the test needs four times as many users.

Solution in brief

Count only users who actually reach the changed experience. Either log the exposure event as late as possible in the code, or filter exposures afterwards to the users for whom control and treatment really differed. Tracing or login logs can also remove bot and internal-service traffic.

Questions this case answers
  • How to recover statistical power in a randomized experiment when most assigned units could not be affected by the change
  • How to compare two model versions online when they agree on most inputs and the effect hides in a small disagreement subset
  • How to find out why an A/B test is underpowered even though the sample looks large?
  • How to remove users who could not see the treatment from an experiment, in code or after the fact?

Tools and methods

  • Power analysis — sample size n=16σ2/Δ2 for 95% confidence and 80% power, used to quantify the cost of dilution.
  • Late exposure logging — log the trigger only where the feature differs between groups.
  • Post-processing filter — drop ineligible users from exposure and metric tables after the test.
  • Distributed tracing and authentication logs — identify traffic that is not a real user.
Continue reading — it's free

Google or a one-time email code. No password.