Problem
A marketplace such as Yandex Market must pair each new seller offer with an existing product card (SKU); both sides number in the hundreds of millions. The pipeline retrieves candidates from two separate vector indexes, text and image, and a CatBoost model scores each pair. Text or image alone often fails: titles differ, or the brand is visible only in a picture. Each missed or wrong match lowers matching quality, measured as recall at fixed precision.
Example from the article: an offer titled "Mantovarka FS3000" (manti cooker) has to be matched to the SKU "Parovarka Braun, belyy" (Braun steamer). The texts barely overlap and the offer shows no brand, yet the match is obvious when both products are seen together.
Solution in brief
Train one multimodal embedding on top of frozen text (BERT) and image (ResNet) encoders. A small Fusion Transformer mixes their vectors, and a contrastive loss runs over the whole batch similarity matrix. Training pairs come from historical matches, hard negatives from the existing pipeline, and labels corrected by a VLM. The embedding serves single-index retrieval and a similarity feature in CatBoost.
Questions this case answers
- How to build one embedding of an item from its text and image so that a single vector index matches new offers to existing catalog entries more accurately
- How to change a contrastive loss so that probability mass is spread across all positive pairs in a batch instead of being assigned row by row
- How to build one multimodal embedding of a product from its text and images
- How to match new marketplace offers to existing catalog products (SKUs) more accurately
Tools and methods
- Frozen BERT and ResNet — existing unimodal encoders
- Fusion Transformer — mixes text and image vectors into a 256-dimensional embedding
- Whole-batch contrastive loss — batch of 10 thousand pairs
- VLM relabeling with a JSON schema — fixes false pairs
- CatBoost — pair classifier that receives
multimodal_similarity
Google or a one-time email code. No password.
