Fine-Tuned CLIP: Better Listing Experience and 80% More Budget-Friendly

Chung An (Andy) Chen23 Dec 2023

Problem

When a seller uploads a photo, Mercari (Japan and US) pre-fills category, brand, title and color for half a million listings a day. A classifier cannot predict them directly: there are more brands and categories than the 5000+ classes of the old InceptionV3 model. So the service embeds the photo, searches an index of 50-100 million item image embeddings, and lets the top-K similar items vote. After a move to a managed vector service, public CLIP variants were only as accurate as the in-house model, and the index could not be scaled down cheaply.

Example from the article: a photo of Nike sneakers retrieves mostly Nike sneakers, which vote "Nike" as brand and "Shoes" as category.

Solution in brief

Fine-tune CLIP on the marketplace's own image-text pairs in two rounds. In the second round, freeze the vision model and train an added linear layer that cuts the output from 512 to 64 dimensions. The index shrinks eightfold with accuracy close to the 512-dimension model.

Questions this case answers
  • How to predict many-valued item attributes from an image when the label space is too large for a classifier, using nearest-neighbor retrieval over a large embedding index
  • How to shrink the embedding dimension of a retrieval index to cut serving cost while keeping prediction accuracy close to the full-size model
  • How to pre-fill brand and category when a seller uploads a photo, if there are far more brands and categories than a classifier can have classes?
  • How to cut the serving cost of a managed vector index of tens of millions of images without losing much accuracy?

Tools and methods

  • CLIP — base image-text model (151 million parameters, 512 dimensions), fine-tuned
  • InfoNCE loss — contrastive loss used in both fine-tuning rounds
  • Linear dimension-reduction layers — 512 to 64 dimensions, added before L2 normalization
  • ScaNN / Vertex AI Vector Search — nearest-neighbor index of image embeddings
  • Top-K voting — brand and category taken from the most similar listings
Continue reading — it's free

Google or a one-time email code. No password.