Problem
When DoorDash expanded from restaurant food into groceries and alcohol, it had to cover hundreds of thousands of SKUs in new categories. Training attribute-extraction and search-relevance models needs many labels, and outsourcing annotation to humans is slow and costly. Catalog records contain errors that nobody can review by hand, and search engagement data is too sparse for rare "tail" queries.
Example from the article: a title says "Honey Smoked" turkey breast, but the package photo says "Oven Roasted", and the LLM flags the record as a mismatch.
Solution in brief
DoorDash uses LLMs as cheap annotators. A few manual "golden" labels seed a retrieval-augmented prompt that produces many "silver" labels, which then train an attribute-extraction model. The same idea covers catalog checks (the LLM compares fields with the photo and assigns a priority) and search relevance (several LLMs vote on whether a result fits the query).
Questions this case answers
- How to get enough labeled examples for a new category or task when human annotation is slow and expensive
- How to detect inconsistencies between structured text fields and an image at catalog scale and rank them by severity
- How to train a relevance model when behavioral signals are noisy and sparse for the long tail of queries
- How to get training labels for product attributes in a new catalog category without paying for a large human annotation round?
Tools and methods
- Golden and silver annotations — a small manual seed set expanded into a large LLM-labeled training set
- Retrieval-augmented generation (RAG) — adds similar labeled examples to the labeling prompt
- Fine-tuned LLM — one attribute-extraction model across categories
- LLM catalog audit — checks item name, unit and photo for contradictions and assigns P0, P1 or P2
- Consensus labeling with LLMs — several LLM judges label search results for relevance
- Ray, LoRA/QLoRA, distillation — scaling LLM inference, fine-tuning and serving
Google or a one-time email code. No password.