BERTScore for the Russian Language

Dari Baturova1 Nov 2024

Problem

Teams that ship Russian text generation (summaries, rewrites, short titles) need a cheap automatic metric to compare model versions. BLEU and ROUGE count word overlap, so a correct paraphrase scores low. LLM judges understand meaning but cost more and carry biases. BERTScore compares contextual embeddings, but its original study covered only English, Chinese and Turkish, and the score depends heavily on which model supplies the embeddings.

Example from the article: for the same reference/generation pair in a simplification task, BERTScore with XLM-RoBERTa-base is 0.94 and with mBART-large-cc25 it is 0.62.

Solution in brief

Compute BERTScore with many candidate encoders, taking embeddings from every layer, not only the last. Correlate each configuration with quality scores over six Russian datasets, and keep the model and layer that correlate best. The speakers got reference scores from two LLMs that grade each other's outputs on a 1–5 scale. The best single configuration is layer 20 of ai-forever/ru-en-RoSBERTa.

Questions this case answers
  • How to choose the embedding model and layer for an automatic similarity metric for generated non-English text when no human scores exist
  • How to get reliable reference quality scores by having language models grade each other's outputs on a 1-5 scale
  • How to choose the embedding model and layer for BERTScore when evaluating generated Russian text
  • How to automatically score summaries, simplifications and short generations in Russian without human raters

Tools and methods

  • BERTScore — similarity of reference and generated text from contextual token embeddings.
  • ru-en-RoSBERTa, ruBert-base, ruBert-large, byt5-base, byt5-large — best encoders found for the embeddings.
  • LLM-as-a-judge with cross-checking — GigaChat Lite grades YandexGPT Lite outputs and vice versa, three prompts per text.
  • Pearson correlation — agreement between BERTScore and the judge score, computed per layer.
  • BLEU, ROUGE-1/2/L — baselines.
Continue reading — it's free

Google or a one-time email code. No password.