Problem
A team generating text with an LLM at scale cannot read the outputs, so it cannot tell which hurt trust. A second LLM can score them, but then nobody checks the scorer: if it drifts and starts approving bad text, nothing downstream notices. Netflix faced this with recommendation explanations, short messages on a title page saying why a title was recommended. It generated hundreds of thousands a week.
Example from the article: a judge correctly failed an explanation pairing two family fantasies that share neither songs nor ensemble cast, but its stated reason was incomplete. A label-only check counts this as agreement.
Solution in brief
Treat the judge as a model with a lifecycle and keep humans in every phase. Build a benchmark with human labels and rationales, tune the rubric until labels and reasons match humans, and use the judge as gate and critic. Compare it weekly with a rater panel; drift triggers re-tuning that a person approves.
Questions this case answers
- How to align an automated text-quality judge with human raters so that it rejects bad outputs for the same reasons humans give
- How to monitor an automated evaluator over time against a noisy human panel and trigger re-tuning when it drifts
- How to check hundreds of thousands of LLM-generated product texts a week when humans cannot read them all?
- How to make an LLM judge agree with human raters on the reason for a rejection, not only on the label?
Tools and methods
- LLM-as-a-judge — one judge per criterion: fixed prompt plus rubric, returns PASS/FAIL and a rationale
- Reasoning-Aligned Rubric Tuning (RART) — rubric edits driven by label and reasoning errors
- Meta-Judge — compares the judge's rationale with a human's
- Reflector — rewrites the rubric from mismatched examples
- Golden set — benchmark with rationales on fails
- Weekly rater panel — three or more raters; majority label is ground truth
Google or a one-time email code. No password.