How to Build Multilingual Product Matching Without Data

Alexander Deev25 May 2024

Problem

Edadil, a discount and retail-offer aggregator owned by Yandex, receives millions of product offers a day. Its "automoderation" models extract segment, category and brand from each offer name and match the offer to a product in the internal SKU database. Russia has years of annotation. A new country (Turkey, Spain, Portugal) has none, annotators are expensive and slow to train, and management wants a launch fast enough to show a payoff.

Example from the article: cheeses from Turkey, Spain and Portugal (e.g. "Taze Sürülebilir Peynir Teksüt 180 G"). An annotator labels category and brand at a glance, but creating the matching SKU takes far longer.

Solution in brief

Reuse the Russian match base: translate both sides of every pair into the target language with Google Translate and train a multilingual bi-encoder plus cross-encoder on the translated pairs. Then distill a Russian teacher model into a multilingual student. A cheap category-brand model and an image model give a first baseline before any of this.

Questions this case answers
  • How to launch matching of items to catalog entries in a new language with no labeled match pairs by reusing a match base from another language
  • How to train a two-stage bi-encoder and cross-encoder matcher on machine-translated pairs and distill it into a multilingual student model
  • How to launch product matching in a new country or language without any labeled matches
  • How to train a multilingual bi-encoder and cross-encoder from an existing matching base in one language

Tools and methods

  • Bi-encoder with batch softmax contrastive (BSC) loss — candidate generator
  • Cross-encoder — re-scores candidates
  • Google Translate — turns the Russian match base into training data for each language
  • paraphrase-multilingual-MiniLM-L12-v2 (Sentence Transformers) — initial weights of both encoders
  • Knowledge distillation (MSE losses) — transfers the Russian model to foreign text
  • Category-brand model, image model (ResNet-based) — baselines
Continue reading — it's free

Google or a one-time email code. No password.