Problem
Netflix shows a synopsis for every title and episode, in many languages and styles, and each must be factual. Manual checking does not scale, so an LLM judge reads all known facts about a title plus a rubric and returns a verdict. It ran five times per synopsis on contexts above 100k tokens, and one failing run failed the synopsis. That was expensive, and the rationales often disagreed with human annotators: the judge failed synopses for nitpicks and missed subtle errors.
Example from the article: a synopsis saying that Bridgerton follows Lady Whistledown and her gossip sheet is literally true, yet it misleads a new viewer, because the show is about Daphne Bridgerton and the marriage market.
Solution in brief
Replace five identical general-purpose judge calls with two specialized judges (literal and implicature). Keep a verdict if the judge is confident. If it is not, extract the judge's own doubts as atomic claims and recheck each claim with a targeted second-stage judge that can overturn the verdict.
Questions this case answers
- How to verify the factuality of generated text at scale when a single long-context reasoning-model judge is expensive and misses subtle errors
- How to cut the cost of LLM-based factuality checks of generated text without losing accuracy?
- How to make an LLM judge catch subtle misleading statements that are literally true?
- How to align an LLM judge's rationales with those of human annotators?
Tools and methods
- LLM-as-a-judge — two specialized reasoning-model judges instead of repeated identical ones
- LLM cascade with confidence gating — only low-confidence verdicts go to the second stage
- Atomic-claim router — turns the judges' doubts into self-contained claims and sends each to a web check or a source-material recheck
- Hallucination detection — factuality is checked against title metadata and text assets
Google or a one-time email code. No password.