Problem
Netflix shows each member a short synopsis of a title. It hosts hundreds of thousands of synopses, usually several variants per show. Good ones help members choose. Poor ones mislead them and push them to stop watching. Only a small team of expert writers can judge quality against the style rubric, so manual review cannot keep up with the catalog.
Example from the article: the team scores each synopsis with an LLM judge on four quality dimensions, among them precision, clarity, tone and factuality, and can find problems weeks or months before a show debuts.
Solution in brief
Build a golden set of binary expert labels, then give each criterion its own LLM judge. Tune it with prompt optimization, rationale summaries, repeated sampling and, for factuality, narrow context-specific agents. Finally check that the combined score tracks take fraction and abandonment.
Questions this case answers
- How to score a large volume of subjective, user-facing text against an expert rubric when expert labels are scarce and experts disagree with each other
- How to make an LLM judge more accurate on hard criteria without making its explanations unreadable or the cost prohibitive
- How to validate an automatic quality score against downstream behavioral metrics when there is no randomized variation in the score
- How to check the quality of hundreds of thousands of generated or written texts when only a few experts can grade them
Tools and methods
- LLM-as-a-judge — one prompt per criterion that returns an explanation and a binary score
- Golden set — about 600 synopses with binary criterion-level expert labels, used to measure judge accuracy
- Automatic Prompt Optimization (APO) — tunes judge prompts on a dev set of about 300 samples
- Tiered rationales and consensus scoring — spend more inference compute on hard criteria
- Agents-as-a-Judge — one narrow factuality agent per type of error, combined by minimum
Google or a one-time email code. No password.