Problem
A team building search or RAG over its own documents has to pick an embedding model, and new ones appear every few months. Public leaderboards rank them, but a model may have been trained on the benchmark datasets, so a high rank can turn into weak results in production. Models also differ in size (from 32M to 7B parameters), context length and pooling, which changes cost and quality.
Example from the article: a listener asks how to evaluate models for a RAG assistant for operators when the open benchmark may be overfitted. The speaker answers that you always need your own benchmark.
Solution in brief
Use the open leaderboard (ruMTEB for Russian) to shortlist models by quality, size and context length. Then rank the shortlist on your own golden set: queries, correct answers and the whole corpus. Confirm with human feedback from real users.
Questions this case answers
- How to choose a text embedding model for retrieval in a non-English language when public benchmark rankings may not reflect results on the own corpus
- How to build an evaluation set for embedding retrieval from the own corpus, queries and correct answers
- How to choose a text embedding model for retrieval or RAG in Russian
- How to read an embedding leaderboard and avoid picking a model overfitted to the benchmark
Tools and methods
- MTEB / ruMTEB — open embedding benchmarks; ruMTEB is the Russian part
- E5, BGE-M3, Jina-v3 — multilingual encoders on XLM-RoBERTa
- FRIDA, BERTA, ru-en-RoSBERTa, USER2 — Russian and Russian-English encoders
- GritLM, NV-Embed, E5-Mistral, Giga-Embeddings — LLM-based embedders (up to 7B)
- CLS pooling, mean pooling — ways to turn token vectors into one text vector
- Golden set — own queries with correct answers, for benchmarking on your domain
Google or a one-time email code. No password.
