Increase A/B Testing Power by Combining Experiments

Zezhong Zhang24 Oct 2022

Problem

eBay experimenters often end up with several separate reports for one hypothesis. They replicate a surprising result, collect samples in several rounds, or run a new design site by site and channel by channel because engineering cannot ship it everywhere in one sprint. Each report alone can be under-powered, so a real effect looks non-significant, yet analysts and product managers still want one combined readout.

Example from the article: a team tested a recommendation change that removes the de-duplication of previously viewed or clicked items on all placements of the View Item page. It was shipped separately to the eBay website and the native app, which gave two experiments, A and B.

Solution in brief

Combine the per-experiment z-statistics with the weighted z-test, a meta-analysis method. Each experiment is weighted by the inverse of its standard error, so precise experiments count more. The combined p-value, lift and confidence interval then come from summary statistics alone, after six eligibility checks confirm the experiments can be merged.

Questions this case answers
  • How to combine summary readouts from several independent randomized experiments on the same hypothesis when each one alone is under-powered
  • How to decide when separately run experiments are safe to merge into one statistical result
  • How to combine the readouts of several independent A/B tests of the same hypothesis into one result with more power?
  • How to get one decision when a feature is tested site by site or channel by channel in separate experiments?

Tools and methods

  • Weighted z-test — combines z-statistics of independent experiments into one p-value, lift and confidence interval
  • Pooled z-test (baseline) — pools all raw samples and runs one z-test, used as the comparison in simulations
  • Touchstone — eBay's experimentation platform that runs the six checks and computes the combined readout
Continue reading — it's free

Google or a one-time email code. No password.