POLLUX: Evaluating the Generative Capabilities of LLMs for Russian

Nikita Martynov, Daniil Astafurov24 May 2025

Problem

A team that ships or chooses an LLM for open-ended generation (texts, advice, role play, style transfer) has no reliable way to say how good the answers are. For Russian, benchmarks are weak, and typologies of tasks and criteria are either tailored to one task or too broad. Expert labeling of one model's answers over the full basket costs about 7,500 (the slide says "7500 tys chch", probably person-hours), so it cannot run on every model version.

Example from the article: Sber's AGI NLP team built POLLUX for Russian: 2,100 hand-written instructions, 11,500 answers from 7 LLMs, 489,375 expert ratings, and two trained judge models.

Solution in brief

Build the benchmark from two typologies: tasks taken from real user requests, and criteria with a general part plus domain and task-specific parts. Experts write the instructions from scratch and label the answers. Then train a small LLM judge on a million synthetic ratings so that expert evaluation can be repeated cheaply.

Questions this case answers
  • How to build a criteria-based benchmark for open-ended generation from real user requests, using typologies of tasks and criteria and a small judge trained on expert ratings
  • How to make expert evaluation of generated text repeatable and cheap enough to run on every model version
  • How to build a criteria-based benchmark for open-ended LLM generation in a non-English language
  • How to derive task and criteria typologies from real user requests

Tools and methods

  • WildChat + BERTopic — cluster real requests into task types.
  • Expert panels with a criteria matrix — derive criteria and label answers.
  • 0/1/2 rubric scale — score each criterion, with a written rubric per formulation.
  • DeepSeek-R1 — generate references and synthetic ratings for judge training.
  • T-lite-it-1.0 (7B), T-pro-it-1.0 (32B) — base models of the POLLUX judges.
Continue reading — it's free

Google or a one-time email code. No password.