Problem
Mercari is a Japanese C2C marketplace. The "Visually Similar Items" block ("Similar Looks") on the item page used 128-dimensional PCA-compressed embeddings from a MobileNet model that was not fine-tuned. Sellers write the titles and take the photos, and the main language is Japanese. The team also wanted shared image embeddings for other uses, such as search and fraud detection.
Illustrative example: a buyer opens a listing with a photo of kids' jeans and expects to see other jeans that look alike.
Solution in brief
Fine-tune a multilingual SigLIP image-text model on about one million Mercari listings (title plus photo), then use its image encoder as the embedding source. Check it offline against logged clicks, compress vectors with PCA, store them in a vector index and serve nearest neighbors. In an A/B test the tap rate rose 1.5x and purchases via the item page rose 14%.
Questions this case answers
- How to improve nearest-neighbor recommendations built on generic image embeddings when the catalog is user-generated, noisy and non-English
- How to evaluate a candidate embedding model offline against logged clicks before committing to an online experiment
- How to improve visually similar item recommendations on a product page of a marketplace with seller-made photos?
- How to fine-tune a vision-language model on our own catalog to get reusable image embeddings?
Tools and methods
- SigLIP (google/siglip-base-patch16-256-multilingual) — base vision-language model; its image encoder produces the embeddings
- PCA — compresses embeddings from 768 to 128 dimensions for latency and storage
- nDCG@k and precision@k — offline evaluation on click logs
- FAISS — index of about 100,000 images for visual inspection
- Vertex AI Vector Search — production nearest-neighbor index
- TensorRT — inference conversion, about 5x faster than PyTorch
Google or a one-time email code. No password.
