Fact-Checking at Scale: Building a More Accurate and Efficient Synopsis Factuality System

Shreya S Ramanujam, Yongce Li, Afshine Amidi, Sina Ghiassian, Benjamin Klein9 Nov 2026

Problem

Netflix shows a synopsis for every title and episode, in many languages and styles, and each must be factual. Manual checking does not scale, so an LLM judge reads all known facts about a title plus a rubric and returns a verdict. It ran five times per synopsis on contexts above 100k tokens, and one failing run failed the synopsis. That was expensive, and the rationales often disagreed with human annotators: the judge failed synopses for nitpicks and missed subtle errors.

Example from the article: a synopsis saying that Bridgerton follows Lady Whistledown and her gossip sheet is literally true, yet it misleads a new viewer, because the show is about Daphne Bridgerton and the marriage market.

Solution in brief

Replace five identical general-purpose judge calls with two specialized judges (literal and implicature). Keep a verdict if the judge is confident. If it is not, extract the judge's own doubts as atomic claims and recheck each claim with a targeted second-stage judge that can overturn the verdict.

Questions this case answers
  • How to verify the factuality of generated text at scale when a single long-context reasoning-model judge is expensive and misses subtle errors
  • How to cut the cost of LLM-based factuality checks of generated text without losing accuracy?
  • How to make an LLM judge catch subtle misleading statements that are literally true?
  • How to align an LLM judge's rationales with those of human annotators?

Tools and methods

  • LLM-as-a-judge — two specialized reasoning-model judges instead of repeated identical ones
  • LLM cascade with confidence gating — only low-confidence verdicts go to the second stage
  • Atomic-claim router — turns the judges' doubts into self-contained claims and sends each to a web check or a source-material recheck
  • Hallucination detection — factuality is checked against title metadata and text assets
Continue reading — it's free

Google or a one-time email code. No password.