Improve Your Next Experiment by Learning Better Proxy Metrics From Past Experiments

Aurélien Bibaut, Winston Chou, Simon Ejdemyr, Nathan Kallus26 Aug 2024

Problem

A/B tests are too short and noisy to measure long-term outcomes such as retention or revenue. Teams therefore track sensitive short-term proxies (click-through rate, engagement, latency) and hope that improving them improves the north star. At Netflix each team picks its own proxies, so metric tradeoffs across areas are unclear. The obvious ways to check a proxy against the north star give biased answers.

Example from the article: a change raises click-through rate, but the test cannot show whether retention moved. User-level correlation between clicks and retention may even be positive while clickbait lowers retention.

Solution in brief

Treat each past experiment as one data point: its true effect on the proxies and on the north star. Regressing the north star effects on the proxy effects works only after removing the bias from correlated measurement noise. Three estimators do this, and total covariance or JIVE suit most cases. The fitted linear model gives a proxy index.

Questions this case answers
  • How to learn a combination of short-term outcomes that predicts a slow, noisy long-term outcome when only weak treatment effects are observed in each of many past randomized experiments
  • How to estimate the covariance of true treatment effects across experiments when measurement noise in the estimated effects is correlated between metrics
  • How to check that a metric that moves in short A/B tests also moves a long-term north star metric?
  • How to combine several short-term metrics into one index that predicts long-term impact?

Tools and methods

  • Total Covariance (TC) estimator — subtracts the scaled measurement-error covariance from the covariance of estimated effects.
  • Jackknife Instrumental Variables Estimation (JIVE) — same target as TC without assuming identical noise covariance across experiments.
  • Limited Information Maximum Likelihood (LIML) — efficient, but only if proxies fully mediate the effect on the north star.
  • Linear model of treatment effects — regresses north star effects on proxy effects across experiments.
Continue reading — it's free

Google or a one-time email code. No password.