Removing outliers for A/B search tests

Christopher Hawke4 Dec 2023

Problem

Bots that run thousands of searches distort A/B test results on a search product. They inflate the tracked-search count of one variant and drag down click-through rate (CTR) and conversion rate (CVR), so a good variant can look bad. Customers then read the wrong trend and may reject a change that works. There are no bot labels, so a rule must separate heavy real users from bots.

Example from the article: a 50/50 test showed 437,591 tracked searches (51.4%) in variant A and 413,440 (48.6%) in variant B. The post puts this at roughly 3%, or about 20,000 searches, off the configured split, and blames bots.

Solution in brief

Count tracked searches per user, take the natural log, and flag a user as an outlier when the user has at least 100 searches and sits more than 7 standard deviations above the mean of the log counts. Exclude flagged users from every metric by default and show how many were removed.

Questions this case answers
  • How to remove automated or abusive traffic from experiment metrics when no labels exist and events per unit follow a heavy-tailed distribution
  • How to set a defensible outlier cutoff for per-user event counts that does not remove legitimate light users
  • How to stop bot traffic from distorting click-through and conversion rates in an A/B test?
  • How to choose a cutoff that separates heavy real users from bots without labeled data?

Tools and methods

  • Log-transform plus standard-deviation cutoff — flags users with an extreme number of searches
  • Minimum search count (100) — protects light users and tiny datasets from false flags
  • Outlier exclusion in the A/B testing dashboard — applies the filter to CTR, CVR and counts by default
Continue reading — it's free

Google or a one-time email code. No password.