Problem
eBay's recall module picks candidate listings for the recommendation block on the View Item page. It used one modality at a time, either the title or the cover image. Title-only recall returns items whose titles look like the seed's but whose pictures are irrelevant or poor, and it confuses a toy car with a real car. Text and image embeddings lived in separate spaces, so no model used both. Listings whose image and title disagree get lower click-through and purchase rates.
Example from the article: for the search term "Sony headphone", some listings have a headphone title but a cover image of something else.
Solution in brief
Concatenate pretrained text and image embeddings of a listing and feed them to a Siamese two-tower model trained to predict co-clicks. A triplet loss with a TransH hyperplane projection pulls the image and title embeddings of one listing together. A second head predicts the probability of no click from the gap between the two embeddings, which scores image-title mismatch.
Questions this case answers
- How to retrieve candidate items for a seed item when a single modality of the item description returns plausible but wrong or low-quality matches
- How to align embeddings from separately trained encoders of different modalities so they can be compared and fused
- How to score the inconsistency between two views of the same entity using only behavioral feedback as the label
- How to recall similar listings when title similarity alone returns items with the wrong or low-quality cover image?
Tools and methods
- Two-tower Siamese model — predicts co-clicks of two listings
- Triplet loss with TransH projection — aligns image and title embeddings
- Mismatch head — scores image-title mismatch
- BERT — base of the title embedding
- ResNet — base of the image embedding
Google or a one-time email code. No password.