Image replacement in Canva designs using reverse image search

Sam Jacobs28 Jan 2025

Problem

When a third-party media partnership expires, Canva must replace that partner's images in every library template. Replacing images by hand is a slow, manual process.

The replacement must look like the original, be safe to use under the company's intellectual property (IP) rules, and come from a library of more than 150 million images. It must also match the frame's aspect ratio and track library changes.

Example from the article: a template contains a photo of an apple on a table in a park. The designer needs another photo of an apple, ideally red, with a similar background and mood.

Solution in brief

Canva built an image-to-image search. Every library image becomes a DINOv2 embedding, picked after a manual comparison of five models. The embeddings sit in an external vector database with metadata filters, and the top 8 neighbors are shown to designers inside the template editor, who pick a suggestion before the design goes to human review.

Questions this case answers
  • How to retrieve the most visually similar items from a very large catalog when text metadata misses visual nuance and no labeled similarity data exists
  • How to choose between candidate embedding models for similarity search when only human judgment of nearest neighbors is available
  • How to find visually similar replacement images for a large media library without manual searching?
  • How to choose an image embedding model when there is no labeled similarity data?

Tools and methods

  • DINOv2 — image embedding model chosen for production
  • CLIP, ViTMAE, DreamSim, CaiT — other candidate embedding models, not chosen
  • GPT-4o description + CLIP text search — tested alternative, the weakest
  • Faiss — in-memory vector index used for the model comparison
  • External third-party vector database — production store with metadata filtering
  • Human-in-the-loop review — a person checks every replacement before republishing
Continue reading — it's free

Google or a one-time email code. No password.