Full talk · edited transcript

POLLUX: Evaluating the Generative Capabilities of LLMs for Russian

Nikita Martynov, Daniil Astafurov24 May 2025

Nikita Martynov (head of AGI NLP, Sber), Daniil Astafurov (principal engineer, AGI NLP development, Sber)

POLLUX is a project of Sber's AGI NLP team for measuring the generative capabilities of large language models in Russian. It includes two LLM-as-Judge models (7B and 32B), a dataset of expert ratings (in the speech "408 thousand", on slide 3 476K, on slide 28: 489,375 pointwise ratings), a typology of generative tasks, and a typology of criteria.

Continue reading — it's free

Google or a one-time email code. No password.