How to Size For Online Experiments With Ratio Metrics

Wenkai Bao28 Mar 2023

Problem

Experimenters at Expedia Group randomize users, but many primary metrics are ratios, such as clicks per visit. Visits from one user are not independent, so the usual variance formula for a mean does not apply and a sample size based on it is wrong. A test sized wrongly either misses real effects or wastes traffic. The team needed a way to size ratio-metric tests so that both the false-positive rate and the power match their nominal values.

Example from the article: sizing a test whose primary metric is clicks per visit (click-through rate), with assignment at the user level.

Solution in brief

Treat the metric as the ratio of two user-level means and approximate its variance with the first-order delta method. The variance depends on the means, variances and covariance of the numerator and denominator. Put that variance into the standard two-sample sample size formula. A simulation on real data with an artificial lift confirms that the nominal power and significance level hold.

Questions this case answers
  • How to determine the sample size of a randomized experiment when the primary metric is a ratio of two sums whose terms are correlated within the randomization unit
  • How to verify by simulation on historical data that a sample size formula holds its nominal power and false-positive rate
  • How to size an A/B test when the primary metric is a ratio of two user-level metrics, such as clicks per visit?
  • How to compute the p-value for a ratio metric when randomization is by user but the metric is defined per visit?

Tools and methods

  • Delta method sampling distribution — approximates the mean and variance of a ratio of two sample means
  • Sample size formula for ratio metrics — gives users per variant from the delta-method variance, the MDE, the significance level and the power
  • Lift-injection simulation on historical data — measures the empirical power and false-positive rate of the sizing procedure
Continue reading — it's free

Google or a one-time email code. No password.