Problem
Shoppers often know the look they want but not the words for it. Text search matches the query string, so each rephrasing returns a different slice of the inventory, and the user keeps retyping. This hurts discovery in style-driven categories such as furniture and home decor, where taste matters more than attributes.
Example from the article: a user wants pillows for a butterscotch couch in a Turkish-themed room. They learn the pattern is called "kilim" and try "orange kilim pillows", "orange throw kilim pillows" and "kilim pillows". None puts the desired item at the top.
Solution in brief
Let the user click a result they like and ask for visually similar items. Each listing gets an embedding learned jointly from its image and title with a CLIP-like contrastive loss. Approximate nearest-neighbor search over those vectors, indexed in the existing search engine, returns the similar items.
Questions this case answers
- How to let users explore a large catalog by taste or style when the preference cannot be expressed in words
- How to produce and keep fresh embeddings for a very large catalog so nearest-neighbor retrieval works in production
- How to let shoppers browse by style when they cannot put a style preference into a text query?
- How to build a 'see visually similar items' feature on top of an existing text search engine?
Tools and methods
- Multimodal embeddings (ResNet-50 image encoder + BERT title encoder) — one vector per listing from image and title.
- Contrastive (matching) loss — trains both encoders on image-title pairs without human labels.
- Approximate nearest-neighbor (ANN) search — retrieves similar items over about 1.7 billion listings.
- Apache Spark and HDFS — item and image downloaders, batch inference.
- Apache Airflow — orchestrates download, inference and indexing in bulk and delta modes.
Google or a one-time email code. No password.