RuModernBERT and USERv2: The Evolution of Russian-Language Encoders

Boris Malashenko24 May 2025

Problem

Teams that build semantic search or text clustering for Russian need an encoder that gives one good vector per text. Classic BERT-style models do not: they were not trained for it, and comparing all pairs through a cross-encoder takes about k2/2 passes for k texts. Most of the Russian encoders compared in the talk are limited to 512 tokens of context. Open paired data for contrastive training is scarce in Russian, unlike English or Chinese.

Example from the article: VK Lab trained USER2-small (34M) and USER2-base (149M) with 8192-token context. USER2-base scores 61.12 on ruMTEB; the best other base model compared scores 59.37.

Solution in brief

Pretrain a ModernBERT-style encoder on Russian, English and code, then turn it into a sentence encoder in three stages. First a short RetroMAE stage, then contrastive training on 652M text pairs, then fine-tuning on about 4M clean pairs with a teacher-filtered loss. Matryoshka training lets you cut the vector to a fraction of its size.

Questions this case answers
  • How to train a sentence encoder that gives one good vector per text and supports long contexts, when open paired training data is scarce
  • How to build a large weakly supervised pair set for contrastive training in a low-resource language and then clean it for the final stage
  • How to train a Russian-language text encoder for semantic search and clustering
  • How to build a large training set of text pairs for a language with little open data

Tools and methods

  • RuModernBERT (35M, 150M) — Russian and English encoder with 8192-token context
  • RetroMAE — short adaptation stage so that pooled vectors carry meaning
  • InfoNCE with prefixes — contrastive training on weakly supervised pairs
  • GistEmbed — InfoNCE variant where a teacher model filters false negatives
  • Matryoshka representation learning (MRL) — loss on vector prefixes, so the embedding can be truncated
  • ruMTEB, MLDR, Encodechka, Russian SuperGLUE — benchmarks used for evaluation
Continue reading — it's free

Google or a one-time email code. No password.