Evaluating Netflix Show Synopses with LLM-as-a-Judge

Gabriela Alessio, Cameron Taylor, Cameron R. Wolfe13 Apr 2026

Problem

Netflix shows each member a short synopsis of a title. It hosts hundreds of thousands of synopses, usually several variants per show. Good ones help members choose. Poor ones mislead them and push them to stop watching. Only a small team of expert writers can judge quality against the style rubric, so manual review cannot keep up with the catalog.

Example from the article: the team scores each synopsis with an LLM judge on four quality dimensions, among them precision, clarity, tone and factuality, and can find problems weeks or months before a show debuts.

Solution in brief

Build a golden set of binary expert labels, then give each criterion its own LLM judge. Tune it with prompt optimization, rationale summaries, repeated sampling and, for factuality, narrow context-specific agents. Finally check that the combined score tracks take fraction and abandonment.

Questions this case answers
  • How to score a large volume of subjective, user-facing text against an expert rubric when expert labels are scarce and experts disagree with each other
  • How to make an LLM judge more accurate on hard criteria without making its explanations unreadable or the cost prohibitive
  • How to validate an automatic quality score against downstream behavioral metrics when there is no randomized variation in the score
  • How to check the quality of hundreds of thousands of generated or written texts when only a few experts can grade them

Tools and methods

  • LLM-as-a-judge — one prompt per criterion that returns an explanation and a binary score
  • Golden set — about 600 synopses with binary criterion-level expert labels, used to measure judge accuracy
  • Automatic Prompt Optimization (APO) — tunes judge prompts on a dev set of about 300 samples
  • Tiered rationales and consensus scoring — spend more inference compute on hard criteria
  • Agents-as-a-Judge — one narrow factuality agent per type of error, combined by minimum
Continue reading — it's free

Google or a one-time email code. No password.