Adaptive Summarization of News Articles Using a Sliding Window and LLM-as-a-Judge

Georgy Nikanorov23 May 2026

Problem

A large publishing house needed automatic summaries of news articles: separate theses, length adapted to the article, business, legal and editorial rules respected. Editors distrust AI and will not re-check output, so the system must either deliver theses usable as is or warn the editors that it cannot. One LLM call over a long article with many rules gave systematic hallucinations, wrong quotes and facts, and conflicting rules. Any small error forced full regeneration.

Example from the article: the article said "426 units of equipment were purchased", the LLM wrote "about 400 units". The article said "In 1927", the thesis said "In the 1920s". With a 20-word limit the model shortens and invents.

Solution in brief

Generate more theses than needed from overlapping windows of the article. Reject any thesis that fails exact-match and fuzzy named-entity checks against the source, or that scores low on BERTScore precision or LLM-based Faithfulness. Then let an LLM only pick and order indices from the verified theses, without rewriting them.

Questions this case answers
  • How to generate faithful abstractive summaries of long documents where any slightly wrong number, date or name makes the output unusable
  • How to select final outputs from pre-verified candidates without letting an LLM rewrite text and without re-verifying after selection
  • How to generate fact-accurate thesis summaries of long articles with an LLM
  • How to automatically filter hallucinated numbers, dates, quotes and names out of LLM output

Tools and methods

  • Sliding-window generation — intermediate theses from overlapping chunks of the article
  • Exact-match and named-entity validation — Levenshtein and n-gram similarity against the source
  • BERTScore precision and LLM-based Faithfulness — thesis-level filters for errors rules cannot catch
  • LLM-as-a-Judge as a selector — returns indices of theses, never rewrites
  • GPT 5.2 mini — best model in the authors' comparison
Continue reading — it's free

Google or a one-time email code. No password.