Problem
Edadil, a discount and retail-offer aggregator owned by Yandex, receives millions of product offers a day. Its "automoderation" models extract segment, category and brand from each offer name and match the offer to a product in the internal SKU database. Russia has years of annotation. A new country (Turkey, Spain, Portugal) has none, annotators are expensive and slow to train, and management wants a launch fast enough to show a payoff.
Example from the article: cheeses from Turkey, Spain and Portugal (e.g. "Taze Sürülebilir Peynir Teksüt 180 G"). An annotator labels category and brand at a glance, but creating the matching SKU takes far longer.
Solution in brief
Reuse the Russian match base: translate both sides of every pair into the target language with Google Translate and train a multilingual bi-encoder plus cross-encoder on the translated pairs. Then distill a Russian teacher model into a multilingual student. A cheap category-brand model and an image model give a first baseline before any of this.
Questions this case answers
- How to launch matching of items to catalog entries in a new language with no labeled match pairs by reusing a match base from another language
- How to train a two-stage bi-encoder and cross-encoder matcher on machine-translated pairs and distill it into a multilingual student model
- How to launch product matching in a new country or language without any labeled matches
- How to train a multilingual bi-encoder and cross-encoder from an existing matching base in one language
Tools and methods
- Bi-encoder with batch softmax contrastive (BSC) loss — candidate generator
- Cross-encoder — re-scores candidates
- Google Translate — turns the Russian match base into training data for each language
- paraphrase-multilingual-MiniLM-L12-v2 (Sentence Transformers) — initial weights of both encoders
- Knowledge distillation (MSE losses) — transfers the Russian model to foreign text
- Category-brand model, image model (ResNet-based) — baselines
Google or a one-time email code. No password.