Problem
Etsy sells handmade and unique items, which are hard to describe in words. Buyers wanted to search the catalog of nearly 100 million listings with a photo from their phone. Embeddings from a plain category classifier return items of the right type but not the right color, material or pattern. Embeddings trained on seller photos also miss on user photos, which are blurry, poorly lit and shot on busy backgrounds.
Example from the article: a query image returns neighbors from the same taxonomy with the classifier embedding, but they do not match the query in color or pattern.
Solution in brief
Fine-tune a pretrained CNN with several classification heads on shared embeddings: categories, color and other attributes, plus a head trained on buyer review photos. Index listing embeddings in an inverted-file ANN index and embed the query photo in real time on a GPU.
Questions this case answers
- How to retrieve visually similar items from a very large catalog when the query image comes from a different distribution than the catalog images
- How to learn one embedding that stays accurate on several label sets when some labels are sparse and the training signal is hard to monitor
- How to build search by photo for a marketplace catalog of about 100 million listings?
- How to train image embeddings that match both the item category and its color, material and pattern?
Tools and methods
- EfficientNet — pretrained CNN backbone, fine-tuned on Etsy data
- Multi-task classification — separate heads per task over shared embedding weights
- Triplet loss — tried first for visual cohesion, then dropped
- IVF approximate nearest-neighbor index — fast lookup over about 100M listing embeddings
- Real-time GPU inference — embeds the query photo fast enough for interactive search
Google or a one-time email code. No password.