DashCLIP: Leveraging multimodal models for generating semantic embeddings

Omkar Gurjar, Kin Sum Liu, Praveen Kolli, Utsaw Kumar, Mandar Rahurkar11 Feb 2026

Problem

DoorDash's ads team ranked sponsored products using categorical and numerical features: store attributes, context and handcrafted aggregates. These carry engagement signals but not what a product is, and say little about a user's tastes. Off-the-shelf image-text models (CLIP, BLIP, FLAVA) fail on short, specific shopping queries. Engagement logs are biased by position and by what the current system chose to show.

Example from the article: DoorDash needed one set of embeddings reusable for retrieval, ranking and relevance across its consumer packaged goods (CPG) catalog.

Solution in brief

Fine-tune a pre-trained image-text model on the product catalog, then train a text-only query encoder to land in the same space as the product embeddings. Supervision for the second step comes from LLM-generated relevance labels, not clicks. The embeddings and a purchase-history embedding then feed the CTR ranking model as extra features.

Questions this case answers
  • How to learn reusable item and query representations from item content that improve both retrieval and ranking when handcrafted categorical features miss semantics
  • How to align a text-only query encoder with a multimodal item encoder when engagement labels are biased and human labels are scarce
  • How to build product and query embeddings that capture catalog semantics for ad retrieval and ranking?
  • How to adapt an off-the-shelf image-text model to a product catalog so it handles short, specific queries?

Tools and methods

  • BLIP-14M — pre-trained checkpoint that initializes the image, text and query encoders
  • Contrastive learning (ITC, ITM, query-catalog loss) — adapts product encoders and aligns queries with products
  • LLM relevance labeling — a fine-tuned GPT model labels 32 million query-product pairs from 700,000 human labels
  • Deep cross network (DCN) — baseline ranking model that receives the embedding features
  • CLIP, FLAVA — off-the-shelf baselines for retrieval comparison
Continue reading — it's free

Google or a one-time email code. No password.