Problem
Teams run A/B tests with several metrics, and the results often conflict: some metrics improve, others stay flat or get worse. Without a written rule, each shipping decision is ad hoc, and the error rates the team believes it has are not the real ones. The usual multiple-testing correction quietly encodes the rule "ship if any metric moves", which is wrong when metrics move in opposite directions.
Example from the article: Spotify's platform must give one recommendation for experiments that mix success, guardrail and quality tests.
Solution in brief
Write the decision rule first, then derive the test levels and power from it. Success metrics use superiority tests with the false-positive rate split over the success metrics only. Guardrails use non-inferiority tests with no false-positive correction, but each is powered at a stricter level so the whole decision keeps its intended power.
Questions this case answers
- How to combine several test results with different roles into one decision while bounding false-positive and false-negative rates
- How to size a test so a decision that needs many simultaneous passes keeps its intended power
- How to turn the results of several A/B test metrics into one ship or no-ship decision with bounded error rates?
- How to correct significance levels when an experiment has both success metrics and guardrail metrics?
Tools and methods
- Decision rule over four metric types — maps all test results to ship / do not ship
- Superiority, non-inferiority and inferiority tests — success, guardrail and deterioration metrics
- Alpha correction for success metrics — divide the false-positive rate by the number of success metrics
- Beta (power) correction for guardrails — raise the power of each test so the joint decision keeps the intended power
- Quality tests — sample ratio mismatch, pre-exposure bias
Google or a one-time email code. No password.