Problem
A large publishing house needed automatic summaries of news articles: separate theses, length adapted to the article, business, legal and editorial rules respected. Editors distrust AI and will not re-check output, so the system must either deliver theses usable as is or warn the editors that it cannot. One LLM call over a long article with many rules gave systematic hallucinations, wrong quotes and facts, and conflicting rules. Any small error forced full regeneration.
Example from the article: the article said "426 units of equipment were purchased", the LLM wrote "about 400 units". The article said "In 1927", the thesis said "In the 1920s". With a 20-word limit the model shortens and invents.
Solution in brief
Generate more theses than needed from overlapping windows of the article. Reject any thesis that fails exact-match and fuzzy named-entity checks against the source, or that scores low on BERTScore precision or LLM-based Faithfulness. Then let an LLM only pick and order indices from the verified theses, without rewriting them.
Questions this case answers
- How to generate faithful abstractive summaries of long documents where any slightly wrong number, date or name makes the output unusable
- How to select final outputs from pre-verified candidates without letting an LLM rewrite text and without re-verifying after selection
- How to generate fact-accurate thesis summaries of long articles with an LLM
- How to automatically filter hallucinated numbers, dates, quotes and names out of LLM output
Tools and methods
- Sliding-window generation — intermediate theses from overlapping chunks of the article
- Exact-match and named-entity validation — Levenshtein and n-gram similarity against the source
- BERTScore precision and LLM-based Faithfulness — thesis-level filters for errors rules cannot catch
- LLM-as-a-Judge as a selector — returns indices of theses, never rewrites
- GPT 5.2 mini — best model in the authors' comparison
Google or a one-time email code. No password.