Problem
A marketplace with visually diverse items wants visual search and visually similar recommendations. Both need an image embedding. Large models are expensive to train and serve. When the query image is embedded online, the model also needs low latency and a small memory footprint. Teams also need a way to judge a model offline during training.
Example from the article: Etsy needed embeddings for visually similar ads on mobile and for its first search-by-image experience, where a buyer's phone photo is embedded online.
Solution in brief
Fine-tune a small pretrained backbone (EfficientNetB0 or EfficientFormer-l3) with several classification heads at once, then drop the heads and keep the embedding. After each epoch, measure nearest-neighbor Recall@5 and @10 on retrieval sets built from same-item photos and ad clicks. An experimental extra set uses images generated from text queries.
Questions this case answers
- How to train compact item representations from images that satisfy a tight online latency budget and a small GPU budget
- How to evaluate a representation model offline during training when the production system has no query logs for the new interaction type
- How to learn image embeddings that are cheap enough to serve online for visually similar recommendations and search by photo
- How to evaluate image embeddings offline after every training epoch without waiting for an A/B test
Tools and methods
- EfficientNetB0 — small CNN backbone, 256-dimensional embedding, the production baseline
- EfficientFormer-l3 — efficient vision transformer, higher accuracy at slightly higher latency
- Multitask classification heads — taxonomy and color labels as supervision for the embedding
- Per-epoch retrieval callback — Recall@K on brute-force nearest neighbor indexes
- Stable Diffusion (token merging, float 16) — turns text queries into image queries for evaluation
Google or a one-time email code. No password.