Beyond Words: How Multimodal Embeddings Elevate eBay's Product Recommendations

Shibo Song, Lili Weng, Shawn Zhou13 Sep 2023

Problem

eBay's recall module picks candidate listings for the recommendation block on the View Item page. It used one modality at a time, either the title or the cover image. Title-only recall returns items whose titles look like the seed's but whose pictures are irrelevant or poor, and it confuses a toy car with a real car. Text and image embeddings lived in separate spaces, so no model used both. Listings whose image and title disagree get lower click-through and purchase rates.

Example from the article: for the search term "Sony headphone", some listings have a headphone title but a cover image of something else.

Solution in brief

Concatenate pretrained text and image embeddings of a listing and feed them to a Siamese two-tower model trained to predict co-clicks. A triplet loss with a TransH hyperplane projection pulls the image and title embeddings of one listing together. A second head predicts the probability of no click from the gap between the two embeddings, which scores image-title mismatch.

Questions this case answers
  • How to retrieve candidate items for a seed item when a single modality of the item description returns plausible but wrong or low-quality matches
  • How to align embeddings from separately trained encoders of different modalities so they can be compared and fused
  • How to score the inconsistency between two views of the same entity using only behavioral feedback as the label
  • How to recall similar listings when title similarity alone returns items with the wrong or low-quality cover image?

Tools and methods

  • Two-tower Siamese model — predicts co-clicks of two listings
  • Triplet loss with TransH projection — aligns image and title embeddings
  • Mismatch head — scores image-title mismatch
  • BERT — base of the title embedding
  • ResNet — base of the image embedding
Continue reading — it's free

Google or a one-time email code. No password.