Problem
DoorDash's ads team ranked sponsored products using categorical and numerical features: store attributes, context and handcrafted aggregates. These carry engagement signals but not what a product is, and say little about a user's tastes. Off-the-shelf image-text models (CLIP, BLIP, FLAVA) fail on short, specific shopping queries. Engagement logs are biased by position and by what the current system chose to show.
Example from the article: DoorDash needed one set of embeddings reusable for retrieval, ranking and relevance across its consumer packaged goods (CPG) catalog.
Solution in brief
Fine-tune a pre-trained image-text model on the product catalog, then train a text-only query encoder to land in the same space as the product embeddings. Supervision for the second step comes from LLM-generated relevance labels, not clicks. The embeddings and a purchase-history embedding then feed the CTR ranking model as extra features.
Questions this case answers
- How to learn reusable item and query representations from item content that improve both retrieval and ranking when handcrafted categorical features miss semantics
- How to align a text-only query encoder with a multimodal item encoder when engagement labels are biased and human labels are scarce
- How to build product and query embeddings that capture catalog semantics for ad retrieval and ranking?
- How to adapt an off-the-shelf image-text model to a product catalog so it handles short, specific queries?
Tools and methods
- BLIP-14M — pre-trained checkpoint that initializes the image, text and query encoders
- Contrastive learning (ITC, ITM, query-catalog loss) — adapts product encoders and aligns queries with products
- LLM relevance labeling — a fine-tuned GPT model labels 32 million query-product pairs from 700,000 human labels
- Deep cross network (DCN) — baseline ranking model that receives the embedding features
- CLIP, FLAVA — off-the-shelf baselines for retrieval comparison
Google or a one-time email code. No password.