Importance of offline evaluation to guide model choice

Tiago Cabo7 Oct 2025

Problem

A classifieds marketplace builds "similar items" recommendations with its own Item2vec model. Open-source text embeddings are now good enough that a custom model is hard to justify, but the MTEB leaderboard lists over 200 of them. The team cannot just take the top one: the model must fit memory and latency limits and cover six languages (Portuguese, Polish, Russian, Bulgarian, Ukrainian, Kazakh) in one model. The team chose to validate candidates offline on its own data instead of picking one for an A/B test.

Example from the article: OLX compared open-source multilingual embeddings on its own interaction data and chose multilingual-e5-small, despite a lower MTEB score than several rivals.

Solution in brief

Shortlist models by the public benchmark and your constraints. Build an offline dataset of item pairs from user interactions and rank the shortlist by recall@5. Fine-tune the winner on the same dataset and add a rerank layer (location, price) before the A/B test.

Questions this case answers
  • How to pick one text-embedding model for item-to-item retrieval across several languages under memory and latency limits, without running an A/B test per candidate
  • How to build a labeled offline relevance set for item-to-item retrieval from implicit interaction logs
  • How to choose an open-source embedding model for similar-item recommendations when hundreds are on the leaderboard?
  • How to build an offline evaluation set for similar items from user interaction logs?

Tools and methods

  • MTEB leaderboard (Hugging Face) — first-pass shortlist of embedding models by benchmark score.
  • Offline pair dataset in BEIR layout — item pairs with a weighted interaction score, used to compare models and to train.
  • multilingual-E5 (multilingual-e5-small) — the model chosen after offline comparison.
  • Sentence Transformers with MultipleNegativesRankingLoss — fine-tuning of the chosen model.
  • Rerank layer — weighted average of normalized location distance and similarity score.
Continue reading — it's free

Google or a one-time email code. No password.