Problem
Teams that ship LLM-based features must pick which variants to test online. At Spotify only about 12% of A/B tests end in a shipped positive result, so weak candidates waste experiment capacity. Automated LLM judges score relevance, coherence or intent faster and cheaper than human annotators. But a judge score is a proxy and can drift away from what users do, so evals and experiments need a defined relationship.
Example from the article: a team builds an LLM judge that flags trust-breaking recommendations, such as an item shared with a user it does not fit, fixes the patterns it finds, and needs to know whether the fix improves long-term engagement.
Solution in brief
Use evals as a funnel stage before the experiment, not as a replacement for it. Evals verify that outputs meet quality standards and discard weak candidates, experiments validate user outcomes and guardrails, and the gap between judge scores and experiment results is used to calibrate the judge for the next cycle.
Questions this case answers
- How to sequence cheap automated quality scoring and controlled experiments so that experiment capacity goes to the best candidates
- How to validate an automated proxy scorer against causal user outcomes and keep it calibrated over time
- How to combine LLM evals with online A/B tests so that only promising treatments reach an experiment?
- How to check that an LLM judge's scores track real user outcomes?
Tools and methods
- LLM-as-a-judge — scores relevance, coherence, tone and intent alignment of test sets and A/B variants
- A/B test — validates that the intended change moves the business outcome
- Guardrail metrics — watch dimensions you do not optimize, such as session length, crash rate and retention
Google or a one-time email code. No password.