Reducing experiment duration with predicted control variates

Kelly McManus25 Nov 2025

Problem

Etsy runs hundreds of online experiments, and each one must run long enough to reach statistical power. Long tests slow down learning and cap how many experiments a team can ship per year. Etsy's CUPED (Controlled-experiment Using Pre-Experiment Data), built on 3 pre-experiment features, cut variance by only 7% on average and ended experiments about 1 day earlier. The linear adjustment left much of the noise in place.

Example from the article: a 10-day experiment can conclude in 7 days once the CUPAC-adjusted metric replaces the raw one.

Solution in brief

Etsy trains a gradient-boosted model on 100+ pre-experiment features to predict the outcome metric, then uses the prediction as the covariate in the usual CUPED regression. This method is called CUPAC (Control Using Predictions as Covariate). It lowered variance by 27% on average and shortened experiments by almost 3 days.

Questions this case answers
  • How to shorten randomized experiments by cutting outcome variance when a linear pre-period covariate removes too little noise
  • How to run model-based variance reduction as a batch job across hundreds of concurrent experiments
  • How to shorten A/B tests without losing statistical power?
  • How to get more variance reduction than CUPED gives on a linear pre-period covariate?

Tools and methods

  • CUPED — linear regression adjustment on pre-experiment data; the baseline.
  • CUPAC — the same adjustment, but the covariate is a model prediction of the outcome.
  • LightGBM — the model that predicts the covariate; chosen over XGBoost for speed at billions of predictions.
  • XGBoost — the first model tried.
  • Apache Airflow — orchestrates the daily batch pipeline.
  • Apache Spark — fits the adjustment regression.
Continue reading — it's free

Google or a one-time email code. No password.