Problem
A marketplace team wants LLMs in product features: help sellers write listings, help support agents, extract structured data from free text. Open-source models work better in English, split Russian text into many tokens (slower inference) and score poorly on the company's own tasks, such as ad moderation and search relevance. The team needs features with a measurable effect and a base model that fits its domain at a bearable cost.
Example from the article: sellers at Avito often write an uninformative description, so a model generates one from the photo, title and parameters. An A/B test showed +1.7% orders with delivery.
Solution in brief
Test each LLM feature online. Pick the base model with an internal benchmark built from the company's own tasks, not public leaderboards. Adapt it by continuing Next Token Prediction on a Russian and in-domain corpus, then swap in a tokenizer trained on the company's data: quality stays about the same and inference gets about 1.5 times faster.
Questions this case answers
- How to choose and adapt an open-source LLM to a non-English domain with an internal benchmark, continued pretraining and a cheaper tokenizer
- How to measure the effect of an LLM generation feature for users with an online experiment
- How to ship LLM features in a classifieds marketplace and measure their effect with an A/B test
- How to adapt an open-source LLM to Russian and to a company's own domain with continual pretraining
Tools and methods
- Continual pretraining of Mistral 7B — Next Token Prediction on a 1.5 TB mix of open Russian corpora, Avito data and English data.
- Tokenizer replacement — a new 32k BPE tokenizer trained on Russian-heavy data; embeddings initialized by averaging.
- Supervised fine-tuning (SFT) — question-answer data.
- Internal benchmark — MMLU in Russian and English plus moderation, review moderation and relevance accuracy on balanced samples.
Google or a one-time email code. No password.