Building text summarization for Russian-language texts

Albina Akhmetgareeva4 Jun 2022

Problem

A team at SberDevices wanted a summarization API for Russian texts of different domains: literature, news, reviews and social media. The product needs abstracts of a paragraph or an article, not headlines. Open Russian data does not fit: targets are short headlines, contexts are long so compression is extreme, and news headlines are clickbait full of clichés that leak into generations. Automatic metrics alone could not tell whether the model summarizes or just copies.

Example from the article: on reviews, the model returned almost the whole source text, reordered, with too many source sentences left in the summary.

Solution in brief

Pick training data whose targets are real abstracts (a private editor-written news set plus the Gazeta dataset), fine-tune ruT5-large and compare it with ready-made mBART and mT5 on ROUGE, BLEU and BERTScore. Then check the best model with Toloka assessors on literacy, substance and coherence, and measure compression and the share of copied text per domain.

Questions this case answers
  • How to evaluate a generative summarization feature when reference summaries are scarce and crowd annotators lose attention on long texts
  • How to choose training targets for a text-generation model when the available open data has the wrong output format
  • How to build a Russian-language summarization service that produces abstracts rather than headlines?
  • How to choose training data for a summarizer when open datasets contain only headlines and clickbait?

Tools and methods

  • ruT5-large — the team's chosen model, fine-tuned on abstract-style targets; beam search is the default generation strategy.
  • mBART, mT5 — ready-made multilingual baselines that need no fine-tuning.
  • ruGPT-3 large — a second in-house model compared on the same metrics.
  • ROUGE, BLEU, BERTScore — automatic metrics; BERTScore added to capture closeness in meaning.
  • Toloka — crowd assessment of generated summaries.
Continue reading — it's free

Google or a one-time email code. No password.